Model matrix /

embeddinggemma-2

Checkpoint embeddinggemma-2/hf-914f7f8. Reference: fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126. Scores are the mean MTEB main score ×100 over the 15 quick-tier tasks; Δ is paired with the reference, task by task.

Quality at 768 dimensions

Variant \ HardwareNVIDIA P100 · 16 GBApple M4 Pro · 24 GB
fp32-googlesentence-transformers66.90reference10,961 prefill tok/ssentence-transformers-torch-cuda st6.1.0-torch2.14.1+cu12666.90+0.00 [+0.00, +0.00]losslesscos 1.00004,793 prefill tok/ssentence-transformers-torch-mps st6.1.0-torch2.14.1
fp32-onnx-communityonnxruntime66.90+0.00 [+0.00, +0.00]losslesscos 1.000012,808 prefill tok/sonnxruntime-cuda ort1.29.0-cuda1266.90+0.00 [+0.00, +0.00]losslesscos 1.00002,654 prefill tok/sonnxruntime-cpu ort1.30.0
bf16-googlesentence-transformersnot run66.93+0.03 [−0.04, +0.12]faithfulcos 1.00006,064 prefill tok/ssentence-transformers-torch-mps st6.1.0-torch2.14.1
q8-onnx-communityonnxruntime66.80−0.10 [−0.38, +0.09]faithfulcos 0.999912,620 prefill tok/sonnxruntime-cuda ort1.29.0-cuda1266.80−0.10 [−0.38, +0.09]faithfulcos 0.99992,756 prefill tok/sonnxruntime-cpu ort1.30.0
q4-onnx-communityonnxruntime66.19−0.71 [−1.01, −0.43]degradedcos 0.980512,530 prefill tok/sonnxruntime-cuda ort1.29.0-cuda1266.19−0.71 [−1.01, −0.43]degradedcos 0.98051,391 prefill tok/sonnxruntime-cpu ort1.30.0

Fidelity compares each run's vectors with the reference's on the same 66,939 texts: lossless (cosine ≥ 0.999 and top-10 neighbor overlap ≥ 0.98), faithful, degraded (cosine < 0.98 or overlap < 0.80), broken.

Output dimension (Matryoshka truncation)

each line is one run: its own vectors truncated, then re-normalized
5961636567128256512768output dimension (vector length kept after truncation, log scale)mean MTEB score ×100 (15 tasks)fp32-google · sentence-transformers · p100:p100: 60.61 at 128fp32-google · sentence-transformers · p100:p100: 64.98 at 256fp32-google · sentence-transformers · p100:p100: 66.43 at 512fp32-google · sentence-transformers · p100:p100: 66.90 at 768bf16-google · sentence-transformers · apple-m:m4-pro: 60.80 at 128bf16-google · sentence-transformers · apple-m:m4-pro: 65.05 at 256bf16-google · sentence-transformers · apple-m:m4-pro: 66.41 at 512bf16-google · sentence-transformers · apple-m:m4-pro: 66.93 at 768fp32-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 60.61 at 128fp32-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 64.98 at 256fp32-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 66.43 at 512fp32-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 66.90 at 768fp32-onnx-community · onnxruntime-cuda · p100:p100: 60.61 at 128fp32-onnx-community · onnxruntime-cuda · p100:p100: 64.98 at 256fp32-onnx-community · onnxruntime-cuda · p100:p100: 66.43 at 512fp32-onnx-community · onnxruntime-cuda · p100:p100: 66.90 at 768fp32-google · sentence-transformers · apple-m:m4-pro: 60.61 at 128fp32-google · sentence-transformers · apple-m:m4-pro: 64.98 at 256fp32-google · sentence-transformers · apple-m:m4-pro: 66.43 at 512fp32-google · sentence-transformers · apple-m:m4-pro: 66.90 at 768q8-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 60.66 at 128q8-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 65.06 at 256q8-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 66.48 at 512q8-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 66.80 at 768q8-onnx-community · onnxruntime-cuda · p100:p100: 60.66 at 128q8-onnx-community · onnxruntime-cuda · p100:p100: 65.06 at 256q8-onnx-community · onnxruntime-cuda · p100:p100: 66.48 at 512q8-onnx-community · onnxruntime-cuda · p100:p100: 66.80 at 768q4-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 60.37 at 128q4-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 64.53 at 256q4-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 66.16 at 512q4-onnx-community · onnxruntime-cpu · apple-m:m4-pro: 66.19 at 768q4-onnx-community · onnxruntime-cuda · p100:p100: 60.37 at 128q4-onnx-community · onnxruntime-cuda · p100:p100: 64.53 at 256q4-onnx-community · onnxruntime-cuda · p100:p100: 66.16 at 512q4-onnx-community · onnxruntime-cuda · p100:p100: 66.19 at 768
fp32-google · sentence-transformers · p100:p100bf16-google · sentence-transformers · apple-m:m4-profp32-onnx-community · onnxruntime-cpu · apple-m:m4-profp32-onnx-community · onnxruntime-cuda · p100:p100fp32-google · sentence-transformers · apple-m:m4-proq8-onnx-community · onnxruntime-cpu · apple-m:m4-proq8-onnx-community · onnxruntime-cuda · p100:p100q4-onnx-community · onnxruntime-cpu · apple-m:m4-proq4-onnx-community · onnxruntime-cuda · p100:p100

Per task at 768 dimensions

MTEB main score ×100; Δ vs reference below
Taskfp32-google · sentence-transformers · p100:p100bf16-google · sentence-transformers · apple-m:m4-profp32-onnx-community · onnxruntime-cpu · apple-m:m4-profp32-onnx-community · onnxruntime-cuda · p100:p100fp32-google · sentence-transformers · apple-m:m4-proq8-onnx-community · onnxruntime-cpu · apple-m:m4-proq8-onnx-community · onnxruntime-cuda · p100:p100q4-onnx-community · onnxruntime-cpu · apple-m:m4-proq4-onnx-community · onnxruntime-cuda · p100:p100
NanoArguAnaRetrieval65.0564.91
−0.14		65.05
+0.00		65.05
+0.00		65.05
+0.00		64.91
−0.14		64.91
−0.14		65.01
−0.04		65.01
−0.04
NanoClimateFeverRetrieval28.0128.34
+0.34		28.01
+0.00		28.01
+0.00		28.01
+0.00		27.97
−0.04		27.97
−0.04		27.28
−0.72		27.28
−0.72
NanoDBPediaRetrieval65.2965.33
+0.03		65.29
+0.00		65.29
+0.00		65.29
+0.00		65.17
−0.12		65.17
−0.12		64.40
−0.90		64.40
−0.90
NanoFEVERRetrieval92.4392.43
+0.00		92.43
+0.00		92.43
+0.00		92.43
+0.00		92.34
−0.09		92.34
−0.09		90.47
−1.97		90.47
−1.97
NanoFiQA2018Retrieval44.5944.46
−0.13		44.59
+0.00		44.59
+0.00		44.59
+0.00		44.75
+0.16		44.75
+0.16		44.45
−0.14		44.45
−0.14
NanoHotpotQARetrieval89.5689.61
+0.05		89.56
+0.00		89.56
+0.00		89.56
+0.00		89.56
+0.00		89.56
+0.00		88.79
−0.77		88.79
−0.77
NanoMSMARCORetrieval57.3957.23
−0.16		57.39
+0.00		57.39
+0.00		57.39
+0.00		55.50
−1.89		55.50
−1.89		56.46
−0.93		56.46
−0.93
NanoNFCorpusRetrieval38.0238.14
+0.11		38.02
+0.00		38.02
+0.00		38.02
+0.00		38.10
+0.08		38.10
+0.08		38.27
+0.24		38.27
+0.24
NanoNQRetrieval78.2178.21
+0.00		78.21
+0.00		78.21
+0.00		78.21
+0.00		78.49
+0.28		78.49
+0.28		76.70
−1.51		76.70
−1.51
NanoQuoraRetrieval95.4695.46
+0.00		95.46
+0.00		95.46
+0.00		95.46
+0.00		95.46
+0.00		95.46
+0.00		95.27
−0.20		95.27
−0.20
NanoSCIDOCSRetrieval37.6037.67
+0.07		37.60
+0.00		37.60
+0.00		37.60
+0.00		37.86
+0.26		37.86
+0.26		36.67
−0.93		36.67
−0.93
NanoSciFactRetrieval88.2288.08
−0.14		88.22
+0.00		88.22
+0.00		88.22
+0.00		88.08
−0.14		88.08
−0.14		86.77
−1.45		86.77
−1.45
NanoTouche2020Retrieval43.8844.28
+0.41		43.88
+0.00		43.88
+0.00		43.88
+0.00		44.07
+0.20		44.07
+0.20		43.42
−0.46		43.42
−0.46
STSBenchmark87.5087.48
−0.02		87.50
+0.00		87.50
+0.00		87.50
+0.00		87.48
−0.01		87.48
−0.01		87.24
−0.26		87.24
−0.26
Banking77Classification92.2692.24
−0.02		92.26
+0.00		92.26
+0.00		92.26
+0.00		92.28
+0.02		92.28
+0.02		91.61
−0.66		91.61
−0.66
Mean66.90
reference		66.93
+0.03 [−0.04, +0.12]		66.90
+0.00 [+0.00, +0.00]		66.90
+0.00 [+0.00, +0.00]		66.90
+0.00 [+0.00, +0.00]		66.80
−0.10 [−0.38, +0.09]		66.80
−0.10 [−0.38, +0.09]		66.19
−0.71 [−1.01, −0.43]		66.19
−0.71 [−1.01, −0.43]

Speed

per hardware class, fidelity-gated

Apple M4 Pro · 24 GB

SubjectPrefill tokens/s (engine)Prefill tokens/s (wall)Texts/sBatch p50Batch p95Fidelity
bf16-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.16,0646,03234.8479 ms2.95 sfaithful
fp32-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.14,7934,76327.5578 ms3.84 slossless
q8-onnx-community·onnxruntime-cpu-ort1.30.02,7562,75015.9969 ms6.34 sfaithful
fp32-onnx-community·onnxruntime-cpu-ort1.30.02,6542,64715.3932 ms7.08 slossless
q4-onnx-community·onnxruntime-cpu-ort1.30.01,3911,3908.02.16 s11.81 sdegraded not ranked for speed

NVIDIA P100 · 16 GB

SubjectPrefill tokens/s (engine)Prefill tokens/s (wall)Texts/sBatch p50Batch p95Fidelity
fp32-onnx-community·onnxruntime-cuda-ort1.29.0-cuda1212,8087,69244.4518 ms1.62 slossless
q8-onnx-community·onnxruntime-cuda-ort1.29.0-cuda1212,6207,68944.4519 ms1.63 sfaithful
fp32-google·sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu12610,9616,97040.3543 ms1.80 sreference
q4-onnx-community·onnxruntime-cuda-ort1.29.0-cuda1212,5307,18341.5541 ms1.63 sdegraded not ranked for speed

Embedding is prefill only: input tokens through the model, no decode. Engine time is the server's own forward passes; wall time adds HTTP and base64 on the client. Batches of 32 texts, longest first, one request at a time. A degraded subject is listed but not ranked for speed: being fast is not a win if the vectors moved.

Subjects