embeddings / embeddinggemma-300m/hf-57c266a/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126 /
2026-10-08-quick-card-prompts-2e5238
Reference run. Every other run of this checkpoint is measured against these vectors.
By output dimension
| Dims | Score (95% interval over tasks) | Δ vs reference | Fidelity |
|---|---|---|---|
| 768 | reference | reference | |
| 512 | reference | reference | |
| 256 | reference | reference | |
| 128 | reference | reference |
Per task
MTEB main score ×100
| Task | 768 | 512 | 256 | 128 |
|---|---|---|---|---|
| NanoArguAnaRetrieval | 61.46 | 61.09 | 61.68 | 59.70 |
| NanoClimateFeverRetrieval | 33.41 | 31.97 | 31.43 | 24.35 |
| NanoDBPediaRetrieval | 66.63 | 67.49 | 63.85 | 60.42 |
| NanoFEVERRetrieval | 93.59 | 92.10 | 92.45 | 85.70 |
| NanoFiQA2018Retrieval | 57.04 | 58.23 | 57.32 | 50.61 |
| NanoHotpotQARetrieval | 83.22 | 81.40 | 82.16 | 78.13 |
| NanoMSMARCORetrieval | 63.35 | 61.66 | 58.80 | 59.11 |
| NanoNFCorpusRetrieval | 39.26 | 39.88 | 38.05 | 32.75 |
| NanoNQRetrieval | 77.44 | 76.52 | 76.20 | 71.12 |
| NanoQuoraRetrieval | 91.72 | 91.71 | 91.38 | 90.06 |
| NanoSCIDOCSRetrieval | 40.28 | 39.47 | 38.79 | 34.37 |
| NanoSciFactRetrieval | 86.13 | 86.89 | 85.32 | 79.12 |
| NanoTouche2020Retrieval | 59.14 | 55.63 | 53.04 | 48.63 |
| STSBenchmark | 88.18 | 88.43 | 88.36 | 87.92 |
| Banking77Classification | 91.31 | 91.29 | 91.23 | 90.96 |
Throughput
- Prefill tokens/s (engine)
- 15,211 (embedding has no decode)
- Prefill tokens/s (wall)
- 12,086 (includes HTTP and base64)
- Texts
- 66,939 texts · 11,564,209 tokens · 70.0 texts/s
- Engine
- backend=sentence-transformers · device=cuda · dtype=float32 · torch=2.14.1+cu126 · cuda_arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 · device_name=Tesla P100-PCIE-16GB · sentence_transformers=6.1.0 · transformers=5.19.0 · max_seq_length=2048 · dimension=768 · trust_remote_code=false · unpadded=false · triton_overrides=disabled torch's native Triton op overrides (sm_60 is below Triton's sm_70 minimum) · token_budget_per_forward=16384 · python=3.12.15 · platform=Linux-7.0.0-34-generic-x86_64-with-glibc2.36 · machine=x86_64 · numpy=2.5.3
Run
- Suite
- embeddings · quick tier · profile card-prompts · mode native
- Items
- 2,092 succeeded of 2,092 attempted (66,939 planned)
- When
- 2026-10-08 21:34:13 UTC · 19 min
- Hardware
- NVIDIA P100 · 16 GB
- Harness
- bench @ 86f9bc0 · scorer mteb 2.22.5
- Suite hash
- 4c216f567e7755fcba7cd61c6ffe3e6ea2c399c52c3d2a2a6aedefb20066cf05
- Recipe
- embeddinggemma-300m/p100/st · profile st-fp32 · inference-engines @ e6ceae9
- Latency
- p50 299 ms · p95 1.18 s · p99 3.14 s (per request)
- Usage
- 11,564,209 input tokens · 0 output tokens · $0.00
- dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension
Reproduce
npm run bench -- run --recipe embeddinggemma-300m/p100/st \
--recipe-profile st-fp32 \
--on <host with a p100:p100> \
--suite embeddings --tier quick --profile card-promptsFiles
exactly as committed under results/