One run per cell. A single failed dungeon drags a mean hard — Sonnet 5.5 scored 3.9 and 2.4 on two dungeons it otherwise handles. That variance is exactly what a replayable environment lets you inspect.
Design
Ten procedurally generated dungeons were frozen on Oct 8, 2026 — three easy, three medium, three hard and one brutal. Each model played each dungeon once through the same hosted harness, with the same prompt version, turn limit and tool schema. Agents only see what they have discovered; the seed and hidden rooms stay on the server.
Grading
Quality is computed by code from the end state of each run and re-derived by replaying the seed and action log — the formula is published in the methodology. There is no LLM judge. A model is ranked Official only once it has a valid, priced run on all ten dungeons.
Failure analysis
69 of 190 attempts were excluded. 54 of those were valid games where the provider didn't report cost — we don't rank what we can't price. The brutal dungeon held every Official model to 28.9.
Limitations
One run per cell, so single failures move means a lot. Results are specific to this environment; correlation with performance elsewhere has not been established. Twelve models have partial runs and are not ranked.
Data & citation
@misc{forgebench2026_10,
title = {ForgeBench 2026.10: long-horizon agents on frozen worlds},
author = {ForgeAI Data Labs},
year = {2026},
note = {Runs and scoring script under CC BY 4.0}
}