typed-decisions / qwen-3.8-27b/cerebras-2026-10/opaque/cerebras /

2026-10-06-quick-greedy-nothink-gateway-json-schema-414e03

provider-opaqueharness had local changes

Accuracy

72.2%

95% 68.1%–75.9% · n=500

Brier ↓

0.247

KL from gold 3.93 · ECE 0.144

Valid decisions

495/500

5× HTTP 502 invalid_provider_output

Throughput

1 request(s) in flight

Decisions/s: 7.5 over 66 s wall clock (client to server)
Engine decisions/s: not reported by this server
Input tok/s: 4,409 wall (290,097 tokens by this model's own count)

By question type

Group	Accuracy (95% interval)	n
choice	71.3%63.6%–78.0%	150
noul	84.7%78.0%–89.6%	150
score	63.5%56.6%–69.9%	200

By workflow

Group	Accuracy (95% interval)	n
agent trace observability	64.8%56.1%–72.6%	125
customer service	68.0%59.4%–75.5%	125
invoice processing	82.4%74.8%–88.1%	125
security incidents	73.6%65.3%–80.5%	125

Run

Subject: qwen-3.8-27b/cerebras-2026-10/opaque/cerebras
Suite: typed-decisions · quick tier · profile greedy-nothink · mode gateway:json-schema
Items: 99 succeeded of 100 attempted (100 planned), 1 failed
When: 2026-10-06 06:27:54 UTC · 8 min
Hardware: provider-hosted
Harness: bench @ 3b03788 (with local changes) · scorer bench/typed-decisions 1
Suite hash: 5995f56de30d9a5e5b8ab8d8342a24025e75a52464b4861e2c1d16ca1fc5c7b7
Latency: p50 572 ms · p95 1.44 s · p99 2.68 s (per request)
Usage: 290,097 input tokens · 10,579 output tokens · $0.30

accuracy counts every dispatched decision; kl_from_gold, brier and ece use valid decisions only
ece uses 10-bin top-label confidence; the dataset card does not define its ECE, so ece is not comparable with the card
intervals treat decisions as independent; each case contributes five related decisions, so true uncertainty is somewhat wider

Reproduce

op run --env-file=benchmarks.env.op -- \
  npm run bench -- run --cloud cerebras:qwen-3.8-27b --via gateway:json-schema \
    --suite typed-decisions --tier quick --profile greedy-nothink

Files

exactly as committed under results/