typed-decisions / jev-latest/typesafe-2026-10/opaque/typesafe /

2026-10-06-quick-greedy-nothink-2bef61

provider-opaqueharness had local changes

Accuracy

74.8%

95% 70.8%–78.4% · n=500

Brier ↓

0.146

KL from gold 1.78 · ECE 0.039

Valid decisions

500/500

no invalid outputs

Throughput

2 request(s) in flight

Decisions/s: 55.6 over 9 s wall clock (client to server)
Engine decisions/s: not reported by this server
Input tok/s: 10,362 wall (93,155 tokens by this model's own count)

By question type

Group	Accuracy (95% interval)	n
choice	74.0%66.4%–80.4%	150
noul	84.0%77.3%–89.0%	150
score	68.5%61.8%–74.5%	200

By workflow

Group	Accuracy (95% interval)	n
agent trace observability	70.4%61.9%–77.7%	125
customer service	80.0%72.1%–86.1%	125
invoice processing	77.6%69.5%–84.0%	125
security incidents	71.2%62.7%–78.4%	125

Run

Subject: jev-latest/typesafe-2026-10/opaque/typesafe
Suite: typed-decisions · quick tier · profile greedy-nothink · mode native
Items: 100 succeeded of 100 attempted (100 planned)
When: 2026-10-06 06:26:58 UTC · 0 min
Hardware: provider-hosted
Harness: bench @ 3b03788 (with local changes) · scorer bench/typed-decisions 1
Suite hash: 5995f56de30d9a5e5b8ab8d8342a24025e75a52464b4861e2c1d16ca1fc5c7b7
Latency: p50 171 ms · p95 233 ms · p99 296 ms (per request)
Usage: 93,155 input tokens · 13,504 output tokens · $0.0037

accuracy counts every dispatched decision; kl_from_gold, brier and ece use valid decisions only
ece uses 10-bin top-label confidence; the dataset card does not define its ECE, so ece is not comparable with the card
intervals treat decisions as independent; each case contributes five related decisions, so true uncertainty is somewhat wider

Reproduce

op run --env-file=benchmarks.env.op -- \
  npm run bench -- run --cloud typesafe:jev-latest \
    --suite typed-decisions --tier quick --profile greedy-nothink

Files

exactly as committed under results/