embeddings
embeddingsscorer: mteb 2.22.5ready
MTEB tasks scored offline from vectors that bench collects through the subject's /v1/embeddings endpoint. Output dimensions 768, 512, 256 and 128 are scored from the same vectors (truncate, then L2-normalize), as Matryoshka truncation works.
quick tier · card-prompts
NanoBEIR (13 small retrieval tasks) + STSBenchmark + Banking77Classification · ranked by mean MTEB main score (full dimension)
| Rank | Subject | Score at 1024 | Δ vs ref | At 768 | At 512 | At 256 | At 128 | Fidelity at 1024 |
|---|---|---|---|---|---|---|---|---|
| 1–15 | fp32-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1Apple M4 Pro · 24 GB | — | 68.81 +0.00 | 68.25 +0.00 | 67.34 +0.00 | 63.53 +0.00 | reference | |
| 1–15 | fp32-google·sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126NVIDIA P100 · 16 GB | — | 68.81 reference | 68.25 reference | 67.34 reference | 63.53 reference | reference | |
| 1–15 | bf16-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1Apple M4 Pro · 24 GB | — | 68.80 −0.01 | 68.17 −0.08 | 67.29 −0.04 | 63.50 −0.03 | reference | |
| 1–15 | bf16-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1Apple M4 Pro · 24 GB | — | 66.93 +0.03 | 66.41 −0.02 | 65.05 +0.08 | 60.80 +0.19 | reference | |
| 1–15 | fp32-onnx-community·onnxruntime-cpu-ort1.30.0Apple M4 Pro · 24 GB | — | 66.90 +0.00 | 66.43 +0.00 | 64.98 +0.00 | 60.61 +0.00 | reference | |
| 1–15 | fp32-onnx-community·onnxruntime-cuda-ort1.29.0-cuda12NVIDIA P100 · 16 GB | — | 66.90 +0.00 | 66.43 +0.00 | 64.98 +0.00 | 60.61 +0.00 | reference | |
| 1–15 | fp32-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1Apple M4 Pro · 24 GBharness had local changes | — | 66.90 +0.00 | 66.43 +0.00 | 64.98 +0.00 | 60.61 +0.00 | reference | |
| 1–15 | fp32-google·sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126NVIDIA P100 · 16 GBharness had local changes | — | 66.90 reference | 66.43 reference | 64.98 reference | 60.61 reference | reference | |
| 1–15 | q8-onnx-community·onnxruntime-cpu-ort1.30.0Apple M4 Pro · 24 GB | — | 66.80 −0.10 | 66.48 +0.05 | 65.06 +0.08 | 60.66 +0.05 | reference | |
| 1–15 | q8-onnx-community·onnxruntime-cuda-ort1.29.0-cuda12NVIDIA P100 · 16 GB | — | 66.80 −0.10 | 66.48 +0.05 | 65.06 +0.08 | 60.66 +0.05 | reference | |
| 1–15 | q4-onnx-community·onnxruntime-cpu-ort1.30.0Apple M4 Pro · 24 GB | — | 66.19 −0.71 | 66.16 −0.27 | 64.53 −0.45 | 60.37 −0.24 | reference | |
| 1–15 | q4-onnx-community·onnxruntime-cuda-ort1.29.0-cuda12NVIDIA P100 · 16 GB | — | 66.19 −0.71 | 66.16 −0.27 | 64.53 −0.45 | 60.37 −0.24 | reference | |
| 1–15 | bf16-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1Apple M4 Pro · 24 GB | +0.02 [−0.03, +0.07] | — | — | — | — | losslesscos 0.9999 · top-10 0.98 | |
| 1–15 | fp32-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1Apple M4 Pro · 24 GB | +0.00 [+0.00, +0.00] | — | — | — | — | losslesscos 1.0000 · top-10 1.00 | |
| 1–15 | fp32-liquidai·sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu126NVIDIA P100 · 16 GB | reference | — | — | — | — | reference |
Scores are the mean MTEB main score ×100 over the tier's tasks; the bar is its 95% interval (hover for values). That interval resamples tasks, so it is wide and every row overlaps: what separates these subjects is the paired Δ vs reference, where a colored value means the interval excludes zero (hover a smaller-dimension Δ for its interval).
Speed · quick tier
per hardware class, fidelity-gated
Apple M4 Pro · 24 GB
| Subject | Prefill tokens/s (engine) | Prefill tokens/s (wall) | Texts/s | Batch p50 | Batch p95 | Fidelity |
|---|---|---|---|---|---|---|
| bf16-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1 | 9,221 | 53.4 | 337 ms | 1.76 s | lossless | |
| fp32-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1 | 7,458 | 43.2 | 379 ms | 2.26 s | lossless | |
| bf16-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1 | 6,181 | 41.9 | 593 ms | 1.89 s | lossless | |
| bf16-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1 | 6,032 | 34.8 | 479 ms | 2.95 s | faithful | |
| fp32-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1 | 5,043 | 34.2 | 654 ms | 2.35 s | lossless | |
| fp32-google·sentence-transformers-torch-mps-st6.1.0-torch2.14.1 | 4,763 | 27.5 | 578 ms | 3.84 s | lossless | |
| q8-onnx-community·onnxruntime-cpu-ort1.30.0 | 2,750 | 15.9 | 969 ms | 6.34 s | faithful | |
| fp32-onnx-community·onnxruntime-cpu-ort1.30.0 | 2,647 | 15.3 | 932 ms | 7.08 s | lossless | |
| q4-onnx-community·onnxruntime-cpu-ort1.30.0 | 1,390 | 8.0 | 2.16 s | 11.81 s | degraded not ranked for speed |
NVIDIA P100 · 16 GB
| Subject | Prefill tokens/s (engine) | Prefill tokens/s (wall) | Texts/s | Batch p50 | Batch p95 | Fidelity |
|---|---|---|---|---|---|---|
| fp32-google·sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126 | 12,086 | 70.0 | 299 ms | 1.18 s | reference | |
| fp32-onnx-community·onnxruntime-cuda-ort1.29.0-cuda12 | 7,692 | 44.4 | 518 ms | 1.62 s | lossless | |
| q8-onnx-community·onnxruntime-cuda-ort1.29.0-cuda12 | 7,689 | 44.4 | 519 ms | 1.63 s | faithful | |
| fp32-google·sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126 | 6,970 | 40.3 | 543 ms | 1.80 s | reference | |
| fp32-liquidai·sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu126 | 5,604 | 38.0 | 639 ms | 1.87 s | reference | |
| q4-onnx-community·onnxruntime-cuda-ort1.29.0-cuda12 | 7,183 | 41.5 | 541 ms | 1.63 s | degraded not ranked for speed |
Embedding is prefill only: input tokens through the model, no decode. Engine time is the server's own forward passes; wall time adds HTTP and base64 on the client. Batches of 32 texts, longest first, one request at a time. A degraded subject is listed but not ranked for speed: being fast is not a win if the vectors moved.