Recursive Self Improvement Kernel Optimization

A benchmark where the reward cannot be faked

I spent this summer building KernelAscent, a benchmark that asks one question. Does verified self-improvement compound. Most claims about self-improving models are hard to check because the reward is soft. GPU kernels are different. A kernel is either correct against an fp32 reference or it is not, and its speed sits against a roofline that physics fixes. Correctness is objective and headroom is an absolute number, so a model cannot fake progress it did not make.

That property is the point. It lets me separate the parts of self-improvement that usually blur together, generation, selection, weight updates, and curriculum, and measure each one under matched compute. I also added a positive control that injects known-correct kernels, so a real null looks different from a broken harness. This post is what the runs taught me.

Five ways to ask the same question

The benchmark runs five tasks, each a stricter test of the same idea.

• Capability. Can a model write one correct, fast kernel in a single shot.
• Weight-RSI. Train an open model on its own correct kernels across rounds and re-score on held-out tasks.
• Procedure-RSI. Freeze the weights and let the model rewrite its own strategy library and archive, so closed models can join.
• Closed to open. A closed frontier model rewrites an open trainee's training harness, so improvement flows through tooling.
• Self-play. The model authors strictly harder tasks and solves them, so difficulty and capability co-evolve. This is the true recursion test.

The loop and its two gates

Every self-improvement run follows the same loop. Propose a kernel, verify it, select the good ones, update, and transfer to the next round. Across 50 probes from 0.5B to 15B the data kept pointing at two gates that decide whether the loop compounds or dies.

A model cannot learn from kernels it never manages to write correctly even once.

Learning one. Most models die at the correctness wall

The first gate is brutal and it catches most models. If a model never emits a correct kernel, its supervised set is empty and its weight drift goes to zero. Nothing to train on, nothing to compound. Across the runs 35 of 50 emitted at least one correct kernel, and only 10 went on to compound.

Learning two. Compounding lives in the middle

Bigger is not simply better here. Small models rarely cross the wall. Mid-scale models from 2 to 8B cross it often and compound most. The largest models cross and drift the most, yet they saturate against the task roofline and stop gaining, because there is little headroom left. Compounding tracks the health of the self-training signal, not raw parameter count.

Learning three. The bottleneck is generation, not knowledge

This one surprised me. A linear probe on the hidden states decodes whether a kernel is correct with an AUC near 0.98. The model represents correctness internally almost perfectly. Yet it emits a correct kernel only a small fraction of the time. The model knows more than it can write. The wall is a generation and decoding problem, not a gap in what the model understands.

Learning four. A probe can pick the winner, within limits

If the model knows internally which kernel is correct, that knowledge should be usable at decode time. Reranking K candidates by the correctness probe recovers much of the gap toward the oracle best-of-K ceiling at equal budget. I keep this honest. The within-task ranking signal is modest, and probe selection does not beat plain verification under matched budget. It reads correctness better than it harvests it.

Learning five. It is recursion, not just more sampling

A fair worry is that self-training only mimics drawing more samples. The comparators say otherwise. On the per-scale speed board, Qwen2.5-Coder-1.5B reaches 0.237 through weight-RSI against 0.182 with best-of-20 sampling, a real gain from training rather than extra draws. DeepSeek-1.3B compounds near +0.26 against a fresh frozen baseline at matched budget. Interrupting the recursion with a checkpoint-frozen producer erases the gain, so the compounding is genuine.

Learning six. Closed models improve their own procedure

Weight-RSI is open-weight only. Procedure-RSI opens the door to API models by freezing the weights and letting the model rewrite its executable research procedure. Frontier models improve their own procedure a lot. Some regress, which is a useful reminder that editing your own strategy can hurt when the edits are not verified.

Learning seven. Self-play buys a curriculum, not co-evolution

The true recursion test is self-play, where the model authors harder tasks and solves them. I decompose the gain into two parts. A frozen author that escalates the curriculum accounts for most of it. Updating the author on top of that, the self-referential signal I call live minus frozen, mostly sits near zero for open models up to 15B. The live author also tends to collapse and stop emitting valid tasks unless I gate it with structured task mutation. Open-endedness looks like a one-time upgrade here, not a compounding force.

The intuition behind it

Start with the wall. Self-improvement needs a gradient, and a gradient needs at least one verified-correct kernel to learn from. A model that never writes a correct kernel has an empty training set, so its weights cannot move. The first unit of signal is binary, and it gates everything after it. That is why the funnel loses most of its models at the same place.

Now the mid-scale sweet spot. Two forces pull in opposite directions. Too small and the model rarely clears the wall, so there is no signal to learn from. Too large and the model already sits near the roofline, so almost no headroom is left to climb and the gradient points nowhere useful. Compounding needs a model capable enough to produce correct kernels yet far enough from the ceiling to keep gaining, which is exactly the 2 to 8B band.

The generation bottleneck follows from the probe. If a linear read of the hidden states separates correct from incorrect at AUC near 0.98, the representation already holds the answer. The loss happens when the model samples that knowledge into tokens one step at a time. That is why picking candidates at decode time helps, and why the real frontier is turning internal correctness into emitted correctness.

Finally, why this counts as recursion. Best-of-k only reweights a fixed distribution, so it cannot move past what the model already samples. Training on your own verified kernels shifts the weight distribution toward the correct region, which changes what the model samples next round, which changes what it can then train on. Freeze the producer and that chain breaks and the gain disappears. Compounding also needs the weights to keep moving without erasing old competence, so a run dies when drift saturates or the model forgets.

What I take away

Verified self-improvement is real but narrow. It is gated first by whether a model can write a correct kernel at all, and then by whether it stays plastic without forgetting. Scale helps you cross the first gate and then stops mattering. The knowledge is already inside the model, so the frontier is generation and selection, not understanding. Build the reward so it cannot be faked, control for the boring explanations, and most loud claims about recursion get quiet fast.

The full benchmark, leaderboards, and figure atlas live at ahmd-mohsin.github.io/KernelAscent.