typed-decisions

decisionready

Hugging Face official benchmark (LocalLLaMA/typed-decisions): one state and five typed System One questions per case; gold answers are probability distributions.

core tier · greedy-nothink

the whole test split; 400 cases · ranked by accuracy

Rank	Subject	Mode	Accuracy (95% interval)	KL ↓	Brier ↓	ECE ↓	Valid	p50	Cost
1–5	jev-latestopaque·typesafe provider-opaqueharness had local changes	native	73.8%71.8%–75.7%	1.52	0.148	0.033	2,000/2,000	154 ms	$0.02
1–6	clefgguf-q3_k_m-bartowski·llamacpp-cuda-b11515	native	71.3%69.2%–73.2%	0.20	0.110	0.017	2,000/2,000	11.42 s	$0.00
1–6	gev-26b-decidegguf-iq4_xs-mradermacher·llamacpp-cuda-b11515	native	71.2%69.1%–73.1%	0.31	0.130	0.024	2,000/2,000	14.65 s	$0.00
1–6	clef-flashmlx-q8-mlx-community·clef-mlx-mlx0.32.3-mlxvlm0.7.4 harness had local changes	native	70.5%68.4%–72.4%	0.21	0.111	0.012	2,000/2,000	7.16 s	$0.00
1–6	clef-flashbnb-nf4-iammrduncan·decide-server-clef-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	70.2%68.1%–72.1%	0.21	0.114	0.021	2,000/2,000	3.95 s	$0.00
2–6	jev-27b-vlgguf-q3_k_m-iammrduncan·llamacpp-cuda-b11370-jev	native	69.7%67.6%–71.7%	0.47	0.161	0.070	2,000/2,000	22.48 s	$0.00
7–10	d1-3bbf16-liquidai·decide-server-d1-torch-mps-tf5.19.0-torch2.14.1	native	64.9%62.8%–67.0%	0.28	0.143	0.036	2,000/2,000	987 ms	$0.00
7–10	d1-3bfp16-liquidai·decide-server-d1-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	64.8%62.6%–66.8%	0.28	0.143	0.042	2,000/2,000	996 ms	$0.00
7–10	d1-3bfp32-liquidai·decide-server-d1-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	64.8%62.6%–66.8%	0.28	0.143	0.041	2,000/2,000	895 ms	$0.00
7–10	d1-3bfp32-liquidai·decide-server-d1-torch-mps-tf5.19.0-torch2.14.1	native	64.8%62.6%–66.8%	0.28	0.143	0.041	2,000/2,000	1.21 s	$0.00
11–14	d1-omni-600mfp32-liquidai·decide-server-d1-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	57.4%55.2%–59.6%	0.35	0.173	0.063	2,000/2,000	355 ms	$0.00
11–14	d1-omni-600mfp32-liquidai·decide-server-d1-torch-mps-tf5.19.0-torch2.14.1	native	57.4%55.2%–59.6%	0.35	0.173	0.063	2,000/2,000	520 ms	$0.00
11–14	d1-omni-600mfp16-liquidai·decide-server-d1-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	57.4%55.2%–59.5%	0.35	0.173	0.062	2,000/2,000	352 ms	$0.00
11–14	d1-omni-600mfp16-liquidai·decide-server-d1-torch-mps-tf5.19.0-torch2.14.1	native	57.4%55.2%–59.5%	0.35	0.173	0.062	2,000/2,000	432 ms	$0.00
15–17	layaauto-amp-convai·decide-server-laya-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	36.0%33.9%–38.1%	0.57	0.315	0.175	2,000/2,000	468 ms	$0.00
15–17	layafp32-convai·decide-server-laya-torch-cuda-tf5.19.0-torch2.14.1-cu126	native	35.9%33.8%–38.0%	0.57	0.315	0.176	2,000/2,000	422 ms	$0.00
15–17	layafp32-convai·decide-server-laya-torch-mps-tf5.19.0-torch2.14.1	native	35.9%33.8%–38.0%	0.57	0.315	0.176	2,000/2,000	538 ms	$0.00

Ranks are bands: rows whose accuracy intervals overlap share a range. Accuracy counts every dispatched decision, so invalid outputs count as wrong; calibration columns use valid decisions only.

quick tier · greedy-nothink

25 cases per workflow · ranked by accuracy

Rank	Subject	Mode	Accuracy (95% interval)	KL ↓	Brier ↓	ECE ↓	Valid	p50	Cost
1–2	jev-latestopaque·typesafe provider-opaqueharness had local changes	native	74.8%70.8%–78.4%	1.78	0.146	0.039	500/500	171 ms	$0.0037
1–2	qwen-3.8-27bopaque·cerebras provider-opaqueharness had local changes	gateway:json-schema	72.2%68.1%–75.9%	3.93	0.247	0.144	495/500	572 ms	$0.30

Ranks are bands: rows whose accuracy intervals overlap share a range. Accuracy counts every dispatched decision, so invalid outputs count as wrong; calibration columns use valid decisions only.