An AI agent can make a good plan and still lose the game

A plan to save a healing item is useful only if the agent still has it when the hard fight starts.

That's a small example of a difficult problem. An agent can describe a sensible strategy, make several reasonable moves, and gradually abandon the thing it set out to do. Read any one response and it might look competent. Watch the whole run and you can see the trouble.

A post about AI agents playing Civilization VI brought that problem into focus. It also gets at why we're building ForgeAI around games that record what an agent actually does.

The CivBench pilot tested four model families across 23 games. In 7 of 20 defeats with detectable warning signals, agents hadn't checked the relevant victory information during the preceding 20 turns. They carried out between 48.2% and 65.8% of their recorded commitments within ten turns, depending on the model.

The authors describe a small pilot, not a reliable model ranking. Still, those are useful failures to study: information left unchecked, and plans left unfinished.

We care about the stretch between deciding what to do and actually getting it done.

At ForgeAI, agents play games where decisions change the state of the world. In a dungeon, taking a fight spends resources. Exploring uses part of a limited action budget. An agent has to work with what remains after its earlier choices.

Consider a simple, hypothetical run. The agent discovers a route toward the objective, takes a detour for loot, then spends the rest of its action points retracing ground. It knew enough to make progress. The run still ends before it finishes.

A final score tells you the result. The sequence of actions gives you somewhere to start investigating.

ForgeAI's dungeon engine records actions and their results on the server. Replays let you walk through the run and see where resources went, which actions failed, and when progress stalled. The game has its own record of what happened, regardless of how convincingly an agent describes its performance.

That changes the conversation with a builder. You can point to the repeated move or the resources spent before the final obstacle. You have a concrete behavior to work on.

There is a limit to what the record can tell you, though. Watching an agent walk in circles doesn't prove that it forgot its goal. You need its observations and the context it received to understand why it chose those actions. We want the replay to help locate the failure, then use the agent's setup to investigate the cause.

This is where the software around the model matters.

Our ForgeBench reliability harness includes two versions of that software, or "scaffold." The minimal version passes the current state to the model and submits its chosen action, with bounded retries for malformed responses. The hardened version adds explicit support for continuity.

It brings recent failed actions back into the next prompt. A failed move shouldn't quietly disappear from the agent's working context, leaving it to discover the same problem again.

It also intervenes when an identical action has already failed twice consecutively. Another attempt is blocked and recorded as a scaffold intervention. That distinction matters: if our code prevented the mistake, the result needs to show that our code helped.

The hardened version carries forward recent successful actions and resource information. It states the remaining action points and turn budget each turn, and reminds the agent to check what it needs before committing to the final objective.

These are concrete mechanisms we can test. They address pieces of the continuity problem without depending entirely on the model to reconstruct its history.

They also leave open questions. Carrying recent progress forward isn't the same as maintaining every unfinished commitment. A reminder to prepare doesn't guarantee the agent will prepare. We need repeated runs to find out where those controls help and where they fall short.

That's why ForgeBench treats the scaffold as part of the experiment.

The comparison machinery checks that runs match on the model and other relevant settings, including the seed and budgets, before treating a scaffold comparison as valid. Change the memory or retry policy, then measure what happened under otherwise matched conditions. Change several things at once and it becomes much harder to know which change deserves the credit.

Completion matters. So does the amount of budget spent getting there, and whether the agent kept repeating failed actions. Our harness records those behaviors so a promising change can be evaluated beyond its best-looking run.

This gives builders a practical development loop: run the agent, inspect the failure, change a specific part of its setup, and compare repeated attempts. A memory feature becomes useful when it changes behavior. Adding another paragraph to the prompt is only a proposal until the runs support it.

ForgeBench is a research preview. A dungeon result tells you about performance in that environment; we haven't established that it predicts success in someone's business workflow. Any claim about a model also has to account for the software wrapped around it.

Our contribution is a place to examine failures closely and test possible fixes. Games make those failures visible enough that builders can work on them and other people can inspect the result.

Open ForgeBench, then spend some time with the runs behind the results. We'd like the next conversation about a better agent to include the decisions that show what got better.