What Makes an Experience Valuable for AI Learning?

Forge is not primarily researching whether AI can learn useful skills by playing games. We're researching whether we can identify, extract and manufacture the kinds of decision-making experiences that teach transferable capabilities. The environment is where the experience happens. The decision is the potential learning material. The capability is what we want to develop.

Think about the kinds of decisions an agent might make:

• Recognizing that its original plan no longer fits new evidence.
• Deciding whether to gather more information or act immediately.
• Allocating limited resources between competing objectives.
• Preserving future options rather than maximizing immediate reward.
• Recovering intelligently after an unsuccessful action.
• Understanding when a strategy should be abandoned.

None of those capabilities inherently belongs to a game. The same underlying decision problem can arise in a Dungeon, a real-time strategy game, a coding agent, a logistics operation or an autonomous business. The circumstances, language and available actions might be completely different, but the underlying problem could be remarkably similar. That's what makes this interesting.

If we can identify what makes those decisions structurally equivalent, we may be able to extract something useful from one environment and teach it to a model that operates in another. Of course, two situations appearing similar doesn't guarantee that learning from one improves performance in the other. But discovering and validating those similarities is precisely the work Forge should undertake.

Rather than treating the source environment as the object of the experiment, we should be investigating what the experiences generated inside it can actually teach. And eventually, whether we can deliberately create better experiences to teach those capabilities more effectively. That last part could be particularly important. If we can learn which experiences actually make models better, we could begin generating experiences based on their learning value, rather than simply collecting more data and hoping it's useful.

From experience to decisions to skills

We need to distinguish three things. An experience is something an agent did. A decision is a choice the agent made under particular conditions. A skill is the reusable ability to make appropriate choices across a family of conditions, including recognizing when a familiar approach would be wrong.

That third step is particularly important. Imagine an agent playing a strategy game and deciding whether to scout an enemy position before attacking. We wouldn't want to extract the lesson "always scout before attacking." Instead, we'd want to understand the underlying relationship between uncertainty, the value of additional information, time pressure and the risks of committing to an action.

There will be situations where scouting is clearly the right decision. There will also be situations where gathering more information is a mistake, perhaps because an opportunity to act is disappearing or because the cost of waiting outweighs the value of the information gained. Both types of experience are important. The skill we're trying to teach isn't scouting. It's understanding when gathering more information is worth the cost. And that capability could be useful in a completely different environment, where there are no scouts, armies or enemy bases.

Consider a Dungeon agent deciding whether to explore further before committing to a dangerous route. A software agent facing an uncertain deployment may have the same fundamental problem: whether to spend time gathering information or commit immediately. We don't want the model to learn that exploring corridors is good, or that delaying deployment is good. We want it to learn when the expected value of additional information outweighs its cost, given the uncertainty, constraints and possible consequences.

Crucially, that requires examples where gathering more information was the right decision and examples where acting immediately was preferable. In other words, the useful training data may be the contrast between decisions under different conditions, rather than any single correct action. That's how we begin to teach judgment rather than imitation.

And it suggests something else. Creating several carefully controlled variations of an important experience could be much more valuable than collecting another thousand ordinary runs.

What are we actually trying to improve?

This leads to what we believe is another important part of Forge's thesis. We've spent a lot of time thinking about extracting important decisions. But there's a further step that could be considerably more valuable: extracting the underlying skills from multiple experiences, then teaching those skills to a model before it specializes in a particular field. That gives Forge a clearer three-stage learning hypothesis.

• Stage 1 — Foundation: Language, knowledge, mathematics and basic reasoning.
• Stage 2 — Transferable skill training: Planning, judgment, adaptation, collaboration, resource allocation, recovery, managing uncertainty and understanding when different approaches are appropriate.
• Stage 3 — Specialization: Accounting, logistics, coding, operations, finance, robotics and other specialist domains.

Stage 2 is where we think Forge has an opportunity. This isn't to suggest that today's foundation models lack these capabilities entirely. They clearly have many of them already. The hypothesis is that deliberately improving these underlying abilities could create a much stronger starting point for specialist training.

Think about how humans learn. Someone learning accounting doesn't just benefit from understanding mathematics and the relevant accounting principles. They also benefit from knowing how to plan, prioritize, deal with uncertainty, recognize mistakes, allocate their time and solve problems. Those aren't accounting-specific skills. They're capabilities that influence how effectively someone works and learns within accounting. And they're useful in many other fields too.

So the question we're interested in is whether we can teach AI models the equivalent of these intermediate skills using decision-making experiences extracted from diverse virtual environments. If that works, there are potentially two important improvements. First, a model might be able to learn a new specialty with substantially less specialist training data. Second, it might ultimately reach a higher level of specialist performance because the underlying abilities it brings to that specialty are stronger. Those are different effects. Faster learning doesn't necessarily mean a higher final level of performance, and we need to measure both.

But the broader idea is that instead of training a model to perform more individual tasks, we could improve the underlying abilities that help it learn and solve new tasks. The first approach can produce a very capable specialist. The second could produce a better foundation for many different specialists.

And what makes this particularly interesting for Forge is that we wouldn't necessarily need to manually decide which skills to teach. We could investigate whether those skills can be discovered from the experiences of agents operating in different environments.

This isn't an entirely new theory

There's already substantial research into learning reusable skills, extracting useful information from agent experiences and selecting better training data. In fact, several important components of the system we're imagining have already been demonstrated, although they haven't been combined into the complete capability-discovery system we're proposing. One particularly relevant area is identifying consequential decisions.

Verified Critical Step Optimization (CSO) investigates finding critical decisions in failed agent trajectories, testing alternative actions through continued execution, and creating verified preference pairs. The paper reports improvements using supervision on just 16% of trajectory steps. That's interesting because it suggests we don't necessarily need to train on every step of an agent's experience to capture much of the useful information.

But identifying an important decision isn't the same as identifying valuable training data. That's where another line of research becomes relevant. LESS, from Princeton, selects training examples using estimates of their influence on a target capability. In reported experiments, a selected 5% of the data could outperform the full training set. This is particularly important for Forge because we're not simply trying to find interesting behavior. We're trying to find the examples that actually cause a model to learn something useful.

There's also research addressing the infrastructure needed to train from these experiences. Agent Lightning, from Microsoft, converts complex, multi-step agent executions into training transitions and supports credit assignment. Minari, TRL and Cleanlab provide existing foundations for organizing offline experience, training models using different objectives, and working with uncertain or noisy labels.

Other research looks more directly at reusable skills. SPiRL learns skill representations from offline experience and uses them to accelerate subsequent reinforcement learning. LISA investigates interpretable skill abstractions that can be combined to perform previously unseen instructions. ExpeL extracts reusable insights from agent successes and failures across tasks.

SkillRL tackles the problem of transforming noisy trajectories into a hierarchical library of reusable behavioral skills. Trace2Skill investigates extracting lessons from collections of trajectories and consolidating them into reusable procedures, with reported transfer across models and some unfamiliar tasks. And STILTs and MAML provide precedents for the broader learning hypothesis: intermediate training can improve subsequent task learning, and models can be explicitly optimized to adapt more rapidly.

These approaches don't all solve the same problem. Some identify important actions. Some select valuable training examples. Some learn reusable behaviors. Others help models adapt more quickly. But together, they give us a substantial foundation to build on.

There's also an important distinction between several of these skill-extraction approaches and what Forge ultimately wants to demonstrate. Some systems improve an agent by providing reusable instructions, memories or procedures at runtime. That's useful, but we're interested in whether extracted skills can also be used as training data to improve the model's weights. Rather than simply giving an agent a collection of lessons it can consult, can we improve its underlying capabilities? And can we then show that those improvements help it learn something new? That's a different and potentially more valuable outcome.

So we don't need to repeatedly return to the generic question of whether skill transfer exists. There's already good reason to believe that learned capabilities can transfer, at least under particular conditions. The more interesting uncertainty is whether Forge can reliably identify, isolate, represent and teach the right skills. And whether we can become increasingly effective at determining which experiences produce those improvements.

The distinction between an important decision and valuable training data

This deserves particular attention because it's easy to conflate several different questions.

• Was the decision consequential? Did the choice materially affect the outcome?
• Can we verify a better alternative? Can we replay or reconstruct the situation, test another action and establish stronger evidence about which decision was preferable?
• What underlying decision structure does it express? What general capability might this experience teach, independently of its original environment?
• Does training on the example actually improve the model? Does the information change the model's behavior in a useful way?
• Does that improvement transfer to other contexts? Can the model apply what it learned when the terminology, tools, environment and specific circumstances are different?

Those five questions aren't equivalent. For example, we could identify a critical action that caused a Dungeon failure, replay the situation with a better action and verify that the alternative produced an improved outcome. That's valuable evidence. But the resulting training example may still teach nothing useful to a different model.

Perhaps its representation is poor. Perhaps the model already understands the lesson. Perhaps the example doesn't capture the conditions under which the lesson applies. Or perhaps it teaches the correct response to one particular situation without developing any more general capability.

This suggests that Forge ultimately needs to learn which evidence makes the best training material, not simply which actions have the biggest consequences. That's potentially a substantial research capability in its own right.

CSO helps us investigate which decisions affect outcomes. LESS helps us investigate which data affects learning. Skill-extraction research helps us investigate how reusable capabilities might be represented.

But none of these methods alone answers the combined question: Which kinds of decisions, extracted from which experiences, improve which capabilities in different contexts? That's the question we have begun to organize Forge's research around.