#1n = 10
91.6
Opus 5.5
Evaluations · ForgeBench
Benchmarks measurewhether a model can.We measure whetherit finishes.
Scores, cost and failure analysis under identical conditions — every number linked to a run you can replay.
ForgeBench 2026.10 · Official · 10 frozen dungeonsFull report →
#2n = 10
85.5
GPT Sol 6.1
#3n = 10
72.6
Sonnet 5.5
Quality per dungeon, easy → brutal. 12 more models have partial runs; we only rank a model once it has n ≥ 10.
A private evaluation
Your models, our worlds,one report.
- Anchored scores0–100 against fixed baselines, per scenario and overall, with sample sizes stated.
- Cost per runTokens and dollars per episode, so quality is always read next to price.
- Failure taxonomyWhy runs ended: invalid moves, loops, give-ups, timeouts — not just that they did.
- Replay linksEvery cell opens the run behind it. Disagree with a number? Replay it.
- Harness comparisonThe same model every-tick vs plan-and-execute, to show what the scaffold adds.
Open suite data
Frozen ForgeBench runs as CSV and JSON, with the script that re-scores them. CC BY 4.0.
Want your model evaluated — privately?
Send the models and the decision you're making. We'll scope a run on our environments or yours.
Contact us →