embeddings / embeddinggemma-300m/hf-57c266a/bf16-google/sentence-transformers-torch-mps-st6.1.0-torch2.14.1 /
2026-10-09-quick-card-prompts-081a23
By output dimension
| Dims | Score (95% interval over tasks) | Δ vs reference | Fidelity |
|---|---|---|---|
| 768 | −0.01 [−0.11, +0.07] | losslesscos 1.0000 · top-10 0.98 | |
| 512 | −0.08 [−0.20, +0.02] | faithfulcos 1.0000 · top-10 0.98 | |
| 256 | −0.04 [−0.17, +0.07] | losslesscos 1.0000 · top-10 0.98 | |
| 128 | −0.03 [−0.20, +0.14] | faithfulcos 1.0000 · top-10 0.98 |
Per task
MTEB main score ×100
| Task | 768 | 512 | 256 | 128 |
|---|---|---|---|---|
| NanoArguAnaRetrieval | 61.52 | 61.09 | 61.63 | 59.73 |
| NanoClimateFeverRetrieval | 33.40 | 32.28 | 31.37 | 24.36 |
| NanoDBPediaRetrieval | 66.84 | 67.31 | 63.88 | 60.61 |
| NanoFEVERRetrieval | 93.59 | 92.10 | 91.71 | 85.56 |
| NanoFiQA2018Retrieval | 57.26 | 57.96 | 56.97 | 50.59 |
| NanoHotpotQARetrieval | 83.40 | 81.40 | 82.16 | 77.81 |
| NanoMSMARCORetrieval | 62.81 | 61.69 | 59.18 | 58.15 |
| NanoNFCorpusRetrieval | 39.05 | 39.83 | 38.29 | 32.75 |
| NanoNQRetrieval | 77.41 | 75.81 | 76.20 | 71.88 |
| NanoQuoraRetrieval | 91.78 | 91.74 | 91.38 | 90.07 |
| NanoSCIDOCSRetrieval | 40.28 | 39.35 | 38.74 | 34.25 |
| NanoSciFactRetrieval | 86.03 | 86.89 | 85.38 | 79.12 |
| NanoTouche2020Retrieval | 59.12 | 55.33 | 52.92 | 48.78 |
| STSBenchmark | 88.18 | 88.44 | 88.37 | 87.94 |
| Banking77Classification | 91.30 | 91.28 | 91.21 | 90.94 |
Throughput
- Prefill tokens/s (engine)
- 9,266 (embedding has no decode)
- Prefill tokens/s (wall)
- 9,221 (includes HTTP and base64)
- Texts
- 66,939 texts · 11,564,209 tokens · 53.4 texts/s
- Engine
- backend=sentence-transformers · device=mps · dtype=bfloat16 · torch=2.14.1 · cuda_arch_list=null · device_name=mps · sentence_transformers=6.1.0 · transformers=5.19.0 · max_seq_length=2048 · dimension=768 · trust_remote_code=false · unpadded=false · triton_overrides=null · token_budget_per_forward=16384 · python=3.12.12 · platform=macOS-26.5.2-arm64-arm-64bit · machine=arm64 · numpy=2.5.3
Run
- Subject
- embeddinggemma-300m/hf-57c266a/bf16-google/sentence-transformers-torch-mps-st6.1.0-torch2.14.1
- Suite
- embeddings · quick tier · profile card-prompts · mode native
- Items
- 2,092 succeeded of 2,092 attempted (66,939 planned)
- When
- 2026-10-09 01:34:54 UTC · 25 min
- Hardware
- Apple M4 Pro · 24 GB
- Harness
- bench @ 2298f0b · scorer mteb 2.22.5
- Suite hash
- 4c216f567e7755fcba7cd61c6ffe3e6ea2c399c52c3d2a2a6aedefb20066cf05
- Recipe
- embeddinggemma-300m/apple-m/st · profile st-bf16 · inference-engines @ 1ef8974
- Latency
- p50 337 ms · p95 1.76 s · p99 4.57 s (per request)
- Usage
- 11,564,209 input tokens · 0 output tokens · $0.00
- dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension
Reproduce
npm run bench -- run --recipe embeddinggemma-300m/apple-m/st \
--recipe-profile st-bf16 \
--on <host with a apple-m:m4-pro> \
--suite embeddings --tier quick --profile card-prompts \
--reference results/embeddinggemma-300m/hf-57c266a/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-08-quick-card-prompts-2e5238Files
exactly as committed under results/