What the right harnessdoes for small models
On a loading-dock shift, small open models scored zero choosing every move. Planning while a harness carried out the moves, they reached 63–82.
We ran frontier, small open and scripted policies on a fixed bank of 12 loading-dock scenarios. Frontier models choosing every move scored 93.7–99.4. Small open models choosing every move mostly scored zero, making invalid moves or running out the clock. Given a plan-and-execute harness, the same small models reached 63–82 at about a third of a cent per shift.
Setup
Each scenario is a 24-tick shift with trucks, doors and crews. Delays cost money; the score maps total cost onto 0 (first-come-first-served, no plan) to 100 (the hindsight optimum). In every-tick mode the model picks each action. In harness mode it writes a plan when something changes and the harness executes it.
Results are specific to this environment. Correlation with performance outside the tested environments has not been established — which is exactly why we build the environment to your task.
What it means
The gap between “can't do the job” and “close to the expert” was the environment's interface, not the model's size. That is the kind of finding a judge-graded benchmark can't produce: the score moved because the end state moved, and every run behind it replays.
Every-tick small-model results are n = 3 per model. The best harness runs are still below the scripted expert (75.6). The hindsight optimum is a reference point, not a ceiling. Results are specific to this environment.
Run this studyon your models.
Same 12 scenarios, same scorer. You get the report, and every run behind it replays.
Contact us →