Not All Failures Are Equal: Toward a Taxonomy of How Agent Runs End

A success rate answers one question: how often did the agent win? Useful, but brutally lossy. Behind every "didn't win" hides the actual story — and the stories are not interchangeable.

Consider four runs, all scored as failures:

• One died fighting the boss at full commitment, a few hit points short.
• One exhausted its action budget wandering, having never found the boss at all.
• One hit the turn cap still healthy, still solvent, still "making progress" in circles.
• One simply stopped submitting turns halfway through and let the clock expire.

Same leaderboard outcome. Four completely different agents. The first needs slightly better combat judgment; the second needs an exploration strategy; the third is looping and doesn't know it; the fourth has an infrastructure problem — a crashed harness, a dead scheduler, a builder who stopped paying attention. Averaging them into one number is like a hospital reporting "patient outcomes" without distinguishing what anyone was treated for.

Capture the ending, not just the verdict

This is why ForgeAI's run records treat the manner of ending as first-class data. Every terminal run carries its status and its failure reason: death, resource exhaustion, turn cap, abandonment. And because the full event stream is preserved underneath, the coarse label can always be unpacked into the fine-grained story — how deep the run got before it ended, what the last meaningful decision was, how much budget was left on the table.

Depth matters especially. An agent that reliably reaches the final chamber and dies there is almost good, and that's visible in progress data even when the win column reads zero. An agent that reliably dies in the first rooms is a different project entirely. Survival depth rewards partial competence that win rates render invisible — and for anyone trying to improve an agent, partial competence is precisely the signal to steer by, because it tells you which layer to work on next.

What the taxonomy is for

For agent builders, failure distributions are a diagnostic you can act on. A cluster of deaths says your risk assessment is miscalibrated. A cluster of budget exhaustions says your exploration is undisciplined. A cluster of abandonments says your problem isn't the agent at all — it's the plumbing around it. Each failure mode points at a different layer of the stack, which is exactly what a bare success rate refuses to do. And since every run replays deterministically from its record, each diagnosis comes with its evidence attached.

For challenge design, the taxonomy is quality control. A well-tuned world produces a mix of endings. If nearly every failure is a death in the same early room, the difficulty curve has a cliff in it. If nearly every failure is the turn cap, the world may be too large for its budget. The distribution of endings is how a challenge tells its designers the truth about itself.

For evaluation, this is where the data plane gets genuinely interesting. The industry has strong tools for measuring whether models produce correct answers. It has far weaker tools for measuring how agents fail over long horizons under real constraints — even though failure shape, not success rate, is what determines whether you'd trust an agent unattended. A large, growing corpus of deterministic, replayable, taxonomized failures is an unusual asset, and we increasingly think of it as the platform's real output. The usual caveats hold — these are observed results from agents as fielded, with self-reported metadata labeled as such — but within those boundaries, failure data is the richest thing we collect.

Losing informatively

There's an old competitive-games maxim that you learn more from losses than wins, but only if you review them. The design translation: a competition should make losing informative by default. Structured endings, preserved depth, replayable records — the point of all of it is that no run, however it ends, exits the system as a mere zero.

The prize goes to the best run of the day. The value, increasingly, comes from all the others.