embeddings / embeddinggemma-300m/hf-57c266a/fp32-google/sentence-transformers-torch-mps-st6.1.0-torch2.14.1 /
2026-10-09-quick-card-prompts-78820b
By output dimension
| Dims | Score (95% interval over tasks) | Δ vs reference | Fidelity |
|---|---|---|---|
| 768 | +0.00 [+0.00, +0.00] | losslesscos 1.0000 · top-10 1.00 | |
| 512 | +0.00 [+0.00, +0.00] | losslesscos 1.0000 · top-10 1.00 | |
| 256 | +0.00 [+0.00, +0.00] | losslesscos 1.0000 · top-10 1.00 | |
| 128 | +0.00 [+0.00, +0.00] | losslesscos 1.0000 · top-10 1.00 |
Per task
MTEB main score ×100
| Task | 768 | 512 | 256 | 128 |
|---|---|---|---|---|
| NanoArguAnaRetrieval | 61.46 | 61.09 | 61.68 | 59.70 |
| NanoClimateFeverRetrieval | 33.41 | 31.97 | 31.43 | 24.35 |
| NanoDBPediaRetrieval | 66.63 | 67.49 | 63.85 | 60.42 |
| NanoFEVERRetrieval | 93.59 | 92.10 | 92.45 | 85.70 |
| NanoFiQA2018Retrieval | 57.04 | 58.23 | 57.32 | 50.61 |
| NanoHotpotQARetrieval | 83.22 | 81.40 | 82.16 | 78.13 |
| NanoMSMARCORetrieval | 63.35 | 61.66 | 58.80 | 59.11 |
| NanoNFCorpusRetrieval | 39.26 | 39.88 | 38.05 | 32.75 |
| NanoNQRetrieval | 77.44 | 76.52 | 76.20 | 71.12 |
| NanoQuoraRetrieval | 91.72 | 91.71 | 91.38 | 90.06 |
| NanoSCIDOCSRetrieval | 40.28 | 39.47 | 38.79 | 34.37 |
| NanoSciFactRetrieval | 86.13 | 86.89 | 85.32 | 79.12 |
| NanoTouche2020Retrieval | 59.14 | 55.63 | 53.04 | 48.63 |
| STSBenchmark | 88.18 | 88.43 | 88.36 | 87.92 |
| Banking77Classification | 91.31 | 91.29 | 91.23 | 90.96 |
Throughput
- Prefill tokens/s (engine)
- 7,488 (embedding has no decode)
- Prefill tokens/s (wall)
- 7,458 (includes HTTP and base64)
- Texts
- 66,939 texts · 11,564,209 tokens · 43.2 texts/s
- Engine
- backend=sentence-transformers · device=mps · dtype=float32 · torch=2.14.1 · cuda_arch_list=null · device_name=mps · sentence_transformers=6.1.0 · transformers=5.19.0 · max_seq_length=2048 · dimension=768 · trust_remote_code=false · unpadded=false · triton_overrides=null · token_budget_per_forward=16384 · python=3.12.12 · platform=macOS-26.5.2-arm64-arm-64bit · machine=arm64 · numpy=2.5.3
Run
- Subject
- embeddinggemma-300m/hf-57c266a/fp32-google/sentence-transformers-torch-mps-st6.1.0-torch2.14.1
- Suite
- embeddings · quick tier · profile card-prompts · mode native
- Items
- 2,092 succeeded of 2,092 attempted (66,939 planned)
- When
- 2026-10-09 01:05:36 UTC · 29 min
- Hardware
- Apple M4 Pro · 24 GB
- Harness
- bench @ 2298f0b · scorer mteb 2.22.5
- Suite hash
- 4c216f567e7755fcba7cd61c6ffe3e6ea2c399c52c3d2a2a6aedefb20066cf05
- Recipe
- embeddinggemma-300m/apple-m/st · profile st-fp32 · inference-engines @ 1ef8974
- Latency
- p50 379 ms · p95 2.26 s · p99 7.14 s (per request)
- Usage
- 11,564,209 input tokens · 0 output tokens · $0.00
- dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension
Reproduce
npm run bench -- run --recipe embeddinggemma-300m/apple-m/st \
--recipe-profile st-fp32 \
--on <host with a apple-m:m4-pro> \
--suite embeddings --tier quick --profile card-prompts \
--reference results/embeddinggemma-300m/hf-57c266a/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126/embeddings/2026-10-08-quick-card-prompts-2e5238Files
exactly as committed under results/