laya
Checkpoint laya/hf-7b928d8. Reference: fp32-convai/decide-server-laya-torch-cuda-tf5.19.0-torch2.14.1-cu126. Scored on typed-decisions, core tier: 400 cases × 5 questions = 2,000 decisions per run. Δ is accuracy minus the reference's, in points.
Accuracy by variant and machine
| Variant \ Hardware | NVIDIA P100 · 16 GB | Apple M4 Pro · 24 GB |
|---|---|---|
| fp32-convaidecide-server-laya | 35.9%referenceBrier 0.315 · CI 33.8%–38.0%24.5 decisions/sdecide-server-laya-torch-cuda tf5.19.0-torch2.14.1+cu126 | 35.9%+0.0 ptsBrier 0.315 · CI 33.8%–38.0%19.9 decisions/sdecide-server-laya-torch-mps tf5.19.0-torch2.14.1 |
| auto-amp-convaidecide-server-laya | 36.0%+0.1 ptsBrier 0.315 · CI 33.9%–38.1%22.7 decisions/sdecide-server-laya-torch-cuda tf5.19.0-torch2.14.1+cu126 | not run |
Accuracy counts every dispatched decision. The interval (CI) is Wilson 95%; differences smaller than the intervals' overlap are not resolved by 2,000 decisions.
All runs
| Variant · machine | Accuracy | KL ↓ | Brier ↓ | ECE ↓ | choice | noul | score | Decisions/s | Input tok/s* | p50 / case | Conformance |
|---|---|---|---|---|---|---|---|---|---|---|---|
| fp32-convai · NVIDIA P100 · 16 GBreference | 0.57 | 0.315 | 0.176 | 28.8% | 48.5% | 31.8% | 24.5 | 7,084 | 422 ms | 49/49 | |
| auto-amp-convai · NVIDIA P100 · 16 GB | 0.57 | 0.315 | 0.175 | 29.2% | 48.5% | 31.8% | 22.7 | 6,575 | 468 ms | 49/49 | |
| fp32-convai · Apple M4 Pro · 24 GB | 0.57 | 0.315 | 0.176 | 28.8% | 48.5% | 31.8% | 19.9 | 5,763 | 538 ms | 49/49 |
Decisions/s is over the run's wall-clock time, client to server, so a remote machine includes the network. Engine-only figures are on each run's page. *Input tok/s counts tokens with each model's own tokenizer and prompt layout (d1 counts the state once per question), so it compares one model across machines, not models with each other. p50 is per case (all five questions); runs kept two requests in flight against servers that answer one at a time, so it includes waiting for the other.