lfm2.5-embedding-350m
Checkpoint lfm2.5-embedding-350m/hf-f35ae2c. Reference: fp32-liquidai/sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu126. Scores are the mean MTEB main score ×100 over the 15 quick-tier tasks; Δ is paired with the reference, task by task.
Quality at 1024 dimensions
| Variant \ Hardware | NVIDIA P100 · 16 GB | Apple M4 Pro · 24 GB |
|---|---|---|
| fp32-liquidaisentence-transformers | 66.06reference6,602 prefill tok/ssentence-transformers-torch-cuda st5.1.1-tf4.56.2-torch2.14.1+cu126 | 66.06+0.00 [+0.00, +0.00]losslesscos 1.00005,060 prefill tok/ssentence-transformers-torch-mps st5.1.1-tf4.56.2-torch2.14.1 |
| bf16-liquidaisentence-transformers | not run | 66.07+0.02 [−0.03, +0.07]losslesscos 0.99996,203 prefill tok/ssentence-transformers-torch-mps st5.1.1-tf4.56.2-torch2.14.1 |
Fidelity compares each run's vectors with the reference's on the same 66,939 texts: lossless (cosine ≥ 0.999 and top-10 neighbor overlap ≥ 0.98), faithful, degraded (cosine < 0.98 or overlap < 0.80), broken.
Output dimension (Matryoshka truncation)
Per task at 1024 dimensions
| Task | fp32-liquidai · sentence-transformers · p100:p100 | bf16-liquidai · sentence-transformers · apple-m:m4-pro | fp32-liquidai · sentence-transformers · apple-m:m4-pro |
|---|---|---|---|
| NanoArguAnaRetrieval | 50.43 | 50.40 −0.03 | 50.43 +0.00 |
| NanoClimateFeverRetrieval | 45.07 | 45.07 +0.00 | 45.07 +0.00 |
| NanoDBPediaRetrieval | 64.70 | 65.01 +0.31 | 64.70 +0.00 |
| NanoFEVERRetrieval | 95.31 | 95.31 +0.00 | 95.31 +0.00 |
| NanoFiQA2018Retrieval | 55.72 | 55.73 +0.01 | 55.72 +0.00 |
| NanoHotpotQARetrieval | 85.23 | 85.23 +0.00 | 85.23 +0.00 |
| NanoMSMARCORetrieval | 64.29 | 64.11 −0.17 | 64.29 +0.00 |
| NanoNFCorpusRetrieval | 36.79 | 36.93 +0.14 | 36.79 +0.00 |
| NanoNQRetrieval | 72.38 | 72.41 +0.03 | 72.38 +0.00 |
| NanoQuoraRetrieval | 89.57 | 89.56 −0.01 | 89.57 +0.00 |
| NanoSCIDOCSRetrieval | 39.20 | 39.12 −0.08 | 39.20 +0.00 |
| NanoSciFactRetrieval | 74.99 | 74.99 +0.00 | 74.99 +0.00 |
| NanoTouche2020Retrieval | 54.14 | 54.21 +0.07 | 54.14 +0.00 |
| STSBenchmark | 80.12 | 80.11 −0.01 | 80.12 +0.00 |
| Banking77Classification | 82.89 | 82.91 +0.02 | 82.89 +0.00 |
| Mean | 66.06 reference | 66.07 +0.02 [−0.03, +0.07] | 66.06 +0.00 [+0.00, +0.00] |
Speed
Apple M4 Pro · 24 GB
| Subject | Prefill tokens/s (engine) | Prefill tokens/s (wall) | Texts/s | Batch p50 | Batch p95 | Fidelity |
|---|---|---|---|---|---|---|
| bf16-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1 | 6,181 | 41.9 | 593 ms | 1.89 s | lossless | |
| fp32-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.1 | 5,043 | 34.2 | 654 ms | 2.35 s | lossless |
NVIDIA P100 · 16 GB
| Subject | Prefill tokens/s (engine) | Prefill tokens/s (wall) | Texts/s | Batch p50 | Batch p95 | Fidelity |
|---|---|---|---|---|---|---|
| fp32-liquidai·sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu126 | 5,604 | 38.0 | 639 ms | 1.87 s | reference |
Embedding is prefill only: input tokens through the model, no decode. Engine time is the server's own forward passes; wall time adds HTTP and base64 on the client. Batches of 32 texts, longest first, one request at a time. A degraded subject is listed but not ranked for speed: being fast is not a win if the vectors moved.
Subjects
- lfm2.5-embedding-350mfp32-liquidai·sentence-transformers-torch-cuda-st5.1.1-tf4.56.2-torch2.14.1-cu1262026-10-08-quick-card-prompts-efe68a
- lfm2.5-embedding-350mbf16-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.12026-10-09-quick-card-prompts-249a86
- lfm2.5-embedding-350mfp32-liquidai·sentence-transformers-torch-mps-st5.1.1-tf4.56.2-torch2.14.12026-10-09-quick-card-prompts-ef2a44