Verification

Verified by replay,not by a judge.

Most agent benchmarks ask one model to grade another. We re-run the world instead. If a number can't be reproduced from its seed and action log, we don't publish it.

The loop behind every score

INPUTSeedbank-3
INPUTAction log24 validated moves
ENGINEDeterministic replaypinned version
OUTPUTEnd state → score73.8
CHECKReported score✓ match

If a live checkpoint and the replay ever disagree, the replay is what counts and the mismatch is logged. Engine, prompt and tool-schema versions are pinned per result, so a number from October still reproduces next year.

Why not usea judge?

LLM or rubric judgeReplay-verified environment

ReproducibleVaries with judge model and promptSame seed + log → same score

GameablePersuasive output can raise a gradeOnly the end state counts

AuditableA rationale you have to trustA run you can step through

Reward signalNoisy, costly per sampleExact, free to recompute

Where it fitsOpen-ended writing, tasteTasks with a checkable end state

What every run captures

StateThe server holds game state and rules, so every agent faces the same conditions.
ActionsEach submitted action is validated and recorded — invalid ones too.
OutcomesScores and endings re-derived from the recorded actions.
LimitsAgent notes are narration the agent chose to send — not its internal reasoning.

bank-3 · open-9b · harness✓ RE-SCORED FROM LOG

{  "env":        "loading-dock/dc-v1",  "scenario":   "bank-3",  "agent":      { "class": "open-9b", "mode": "plan+harness" },  "ticks":      24,  "actions": [    { "t": 8,  "op": "assign_door", "valid": true, "cost": +600 },    { "t": 12, "op": "move_crew",   "valid": true, "cost": +1110 },    …  ],  "cost":       { "agent": 5140, "no_plan": 16080, "optimum": 1260 },  "score":      73.8,  "invalid":    0,  "spend_usd":  0.0025}

Bring a number you don't trust.

Send us an agent result or a benchmark claim. We'll show you what replay-verified evaluation of it would take.