embeddings / embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126 /
2026-10-07-quick-card-prompts-d0777f
harness had local changesReference run. Every other run of this checkpoint is measured against these vectors.
By output dimension
| Dims | Score (95% interval over tasks) | Δ vs reference | Fidelity |
|---|---|---|---|
| 768 | reference | reference | |
| 512 | reference | reference | |
| 256 | reference | reference | |
| 128 | reference | reference |
Per task
MTEB main score ×100
| Task | 768 | 512 | 256 | 128 |
|---|---|---|---|---|
| NanoArguAnaRetrieval | 65.05 | 63.24 | 60.19 | 56.78 |
| NanoClimateFeverRetrieval | 28.01 | 28.42 | 27.57 | 27.11 |
| NanoDBPediaRetrieval | 65.29 | 64.04 | 63.44 | 57.99 |
| NanoFEVERRetrieval | 92.43 | 93.22 | 91.10 | 86.85 |
| NanoFiQA2018Retrieval | 44.59 | 43.72 | 42.54 | 32.73 |
| NanoHotpotQARetrieval | 89.56 | 88.38 | 87.62 | 78.12 |
| NanoMSMARCORetrieval | 57.39 | 55.13 | 54.54 | 50.24 |
| NanoNFCorpusRetrieval | 38.02 | 37.08 | 35.19 | 30.34 |
| NanoNQRetrieval | 78.21 | 76.67 | 72.88 | 71.42 |
| NanoQuoraRetrieval | 95.46 | 95.45 | 94.81 | 92.88 |
| NanoSCIDOCSRetrieval | 37.60 | 37.70 | 35.46 | 30.75 |
| NanoSciFactRetrieval | 88.22 | 89.70 | 86.55 | 77.69 |
| NanoTouche2020Retrieval | 43.88 | 43.88 | 43.31 | 38.12 |
| STSBenchmark | 87.50 | 87.55 | 87.38 | 86.90 |
| Banking77Classification | 92.26 | 92.27 | 92.08 | 91.25 |
Throughput
- Prefill tokens/s (engine)
- 10,961 (embedding has no decode)
- Prefill tokens/s (wall)
- 6,970 (includes HTTP and base64)
- Texts
- 66,939 texts · 11,589,994 tokens · 40.3 texts/s
- Engine
- backend=sentence-transformers · device=cuda · dtype=float32 · torch=2.14.1+cu126 · cuda_arch_list=sm_50,sm_60,sm_70,sm_75,sm_80,sm_86,sm_90 · device_name=Tesla P100-PCIE-16GB · sentence_transformers=6.1.0 · transformers=5.19.0 · max_seq_length=8192 · token_budget_per_forward=16384 · python=3.12.15 · platform=Linux-7.0.0-34-generic-x86_64-with-glibc2.36 · machine=x86_64 · numpy=2.5.3
Run
- Subject
- embeddinggemma-2/hf-914f7f8/fp32-google/sentence-transformers-torch-cuda-st6.1.0-torch2.14.1-cu126
- Suite
- embeddings · quick tier · profile card-prompts · mode native
- Items
- 2,092 succeeded of 2,092 attempted (66,939 planned)
- When
- 2026-10-07 03:11:32 UTC · 31 min
- Hardware
- NVIDIA P100 · 16 GB
2026-10-07: device facts copied from the operator's inventory after the run; the launcher did not record them yet - Harness
- bench @ c62a98f (with local changes) · scorer mteb 2.22.5
- Suite hash
- d499aab0d7091e3ba5b211e1cbaedf79ed4c9c86a7214156946d2ee4d21299ec
- Recipe
- embeddinggemma-2/p100/st-ort · profile st-fp32 · inference-engines @ 0ddc734
- Latency
- p50 543 ms · p95 1.80 s · p99 4.56 s (per request)
- Usage
- 11,589,994 input tokens · 0 output tokens · $0.00
- dimensions below 768 are the same vectors truncated and re-normalized (Matryoshka); fidelity compares each dimension with the reference at the same dimension
Reproduce
npm run bench -- run --recipe embeddinggemma-2/p100/st-ort \
--recipe-profile st-fp32 \
--on <host with a p100:p100> \
--suite embeddings --tier quick --profile card-promptsFiles
exactly as committed under results/