Recursive Self Improvement Hack · Evals · SF Tech Week
So we pointed the loop at the evaluator. An LLM writes test suites, we score each suite by how many deliberately planted bugs it catches, and a rewrite loop hardens the suite. Then we asked the question the event poses: do better evals make a coding agent better? Below is what the run actually shows, including the part that did not go the way we pitched it.
And yet
Step through the mechanism. The left loop improves the evaluator; the right loop is a normal agent improving its code against whatever evaluator it is given.
| Role | Model | What it sees |
|---|---|---|
Each mutant is the correct reference with one small, deliberate change. Pick a mutant, then switch between the starting suite and the suite after the rewrite loop. Training mutants are what the rewriter is shown; held-out mutants come from a different generator and are never shown to it.
The rewriter only ever sees surviving training mutants. The held-out lines are the honest ones: bugs the loop never saw. If the suite were memorizing, they would stay flat while training climbed.
These charts follow the primary agent, the primary agent; every agent is in the table below. The agent writes a solution, gets failing tests back, revises, and stops as soon as its suite passes. Dashed lines are what the suite says; solid lines are the hidden oracle. The padded arm is a size-matched control: the starting suite topped up with random reference-labelled cases to the improved suite’s size.
The same suites, handed to several agent models. Correctness is the hidden oracle’s verdict on the final solution; a false pass is a run whose suite said “done” while the code was wrong; bug catch is the share of that agent’s wrong first attempts the suite rejects. The last two columns are paired by unit (as in the stats section) with a 95% CI.
| Agent | First try | Final correctstart · padded · improved · oracle | False passstart → improved | Bug catchstart → improved | Improved − start | Oracle − start |
|---|---|---|---|---|---|---|
Paired comparisons for the primary agent, by unit (closely related tasks, such as two variants of the same spec, form one unit; correctness averaged over seeds): mean difference in true correctness, a 95% confidence interval, and a sign test on per-unit wins and losses. Each block restricts to a different set of units.
| Contrast | Mean diff | 95% CI | Sign test p | Wins / losses |
|---|---|---|---|---|
Held-out catch rate for each problem before and after the rewrite loop, false-alarm rates, and whether the primary agent’s final solution was truly correct under each kind of feedback (fractions are over seeds). Click a column to sort.
Mutants are split into train and held-out sets before the loop starts. The rewriter only sees surviving training mutants. LLM-written held-out mutants come from a different model than the training ones, so the suite cannot learn one generator’s habits.
Agent correctness is not judged by any LLM-written suite. It is differential testing against the hidden reference on a fixed set of random inputs per task. No model sees the reference or the full input set; the “oracle feedback” arm, which exists only as a ceiling, is handed failing cases from it.
A test that fails a correct solution (a wrong expected value) sends the agent chasing a ghost. Suite score is catch rate minus a multiple of the false-alarm rate. Mutants that turn out equivalent to the reference are removed from scoring.
On luhn_complete the improved suite contains a valid case, luhn_complete("\t1234") → ValueError, because the spec allows only spaces. Agents that strip all whitespace fail it. The random-input oracle never generates a tab, so it scores those solutions correct. Measured correctness is therefore an upper bound, and the improved suite’s honesty advantage is slightly understated. This is also why the improved arm’s eval-pass can sit below its measured correctness. The case also drove real fixes: in this run, most agents shown it went on to handle tabs correctly, a fix the oracle could never have asked for.
The rewriter sees each surviving mutant as a diff against the reference. That is more than a practitioner without a reference solution would have, so the suite gains here are an upper bound on what a black-box rewrite loop could get.
The specs, reference solutions and mutants are published on this page. They may end up in future training data, so this benchmark should not be treated as uncontaminated for later runs.
Tasks are small single-function Python problems with crisp specs; this is not SWE-bench. Mutation catch rate is a proxy for suite strength, and planted bugs need not resemble the bugs an agent writes, which is why the agent’s own wrong attempts are reported separately. Results come from a few seeds per task; differences whose confidence interval spans zero are noise.