Post·August 2026·9 min read

The Answers Reinforcement Learning Learns to Forget

RLVR raises pass@1 but narrows reasoning coverage. A base-anchored, off-policy support floor preserves the modes GRPO cannot protect, and turns that coverage into a higher continued-RL ceiling.

The trade nobody agreed to

Reinforcement learning with verifiable rewards makes math models better on their first try. Train Qwen2.5-Math with GRPO and pass@1 climbs. Sample the same model many times and the story changes. The base model, the one before any RL, starts solving problems the trained model can no longer solve. Past a large enough sampling budget the base overtakes the RL model. Training bought single-sample accuracy and paid for it with reasoning coverage.

Figure 1. OlympiadBench pass@k on a released full-RL checkpoint (Oat-Zero-7B) against its base, 572 problems, 1024 samples each. Oat-Zero leads at small k from sharper single samples. The base overtakes near k ≈ 128 and the gap widens to k = 1024. The trade is a crossover, not a free lunch.

The trade is not uniform across difficulty. On easy problems both models saturate. On the hardest problems the RL model stays ahead because the base has no samplable solutions left to lose. The crossover lives in a narrow middle band, problems the base solves at large budget but rarely at small budget. We call this the fragile band.

Figure 2. pass@256, base against Oat-Zero-7B, across difficulty. The base wins only at intermediate difficulty (OlympiadBench). Easy saturates, hard has no base tail to recover. The crossover is a difficulty resonance.

Reinforcement learning does not add reasoning. It sharpens the reasoning a model already samples and buries the reasoning it does not.

Why reward shaping cannot fix this

The resource to track is not entropy. It is finite-budget recoverability. A problem is recoverable at budget K when at least one of K samples is correct. For a reasoning mode M with probability p_θ(M) under the policy, recoverability is

R_K(M) = 1 − (1 − p_θ(M))^K.

Once p_θ(M) falls below about 1/K the mode is gone at that budget. GRPO cannot protect it. Take a correct reasoning mode y for a prompt q. The expected gradient GRPO routes to y scales with how often the policy samples y.

‖ E[g_y] ‖ = O( p_θ(y | q) ).

The factor appears because y contributes to the objective only when it is sampled, and it is sampled with probability p_θ(y | q). As RL sharpens the policy onto its dominant mode, rare correct modes drop below the 1/K sampling threshold. Their gradient goes to zero. They receive no signal and cannot come back.

Reward shaping does not escape the factor. UCPO adds a diversity bonus. PKPO transforms the pass@k reward. RiskPO reweights by risk. Each one multiplies the advantage by a bounded factor, and a bounded factor leaves the O(p_θ) term intact. A method can only push on a mode it samples, so no reward reshaping reaches a mode rarer than 1/K. The modes RL loses are exactly the ones it has stopped sampling.

The fix, a base-anchored support floor

We fix coverage with a constraint instead of a reward. Offline we run the base model on the fragile band and collect a bank of base-correct traces. For each trace we store its base log-probability. During RL we require the policy to keep every banked trace within a fixed factor of its base probability.

log π_θ(y_q | q) ≥ log π_0(y_q | q) + log α, with α = 0.5.

At α = 0.5 the policy may not drop a banked trace below half of the base probability on that trace. Three properties separate this from every reward method. First, it is off-policy. We evaluate the banked trace by teacher forcing, so the correction gradient does not shrink as the policy stops sampling the mode. Second, it constrains log π_θ directly rather than adding a reward, so GRPO's group-relative advantage cannot cancel it. Third, it is one-sided, a ratchet. Only dropping below the floor is penalized, so the floor never drags the policy back toward base and adds no intrinsic pass@1 tax. Where the reward gradient vanishes, the floor gradient stays Ω(1).

We ship two forms. expSR adds a one-sided penalty to the GRPO loss. expPROJ runs projected-gradient correction steps after each GRPO step to restore feasibility. A survival diagnostic logs the fraction of banked base-correct modes still above their floor during training.

What the floor does to reasoning modes

First the mechanism, measured directly. On the 49 base-correct traces for OlympiadBench problems that Oat-Zero lost, RL suppressed every one. Median trace log-probability fell from −254 to −320, all 49 traces dropped, mean 84 nats below base. The pattern replicates on Omni-MATH, 150 of 150 traces suppressed, mean 75 nats. RL does not merely fail these problems. It drives every recoverable solution far under the sampling floor, which is why large-k sampling never gets them back.

Now the causal test. We forked from a single base model and ran identical GRPO for identical steps, with the floor as the only difference. On a held-out bank of 11,664 mode witnesses, plain GRPO collapses 39% of base reasoning modes, meaning it drives them more than 10 nats below base. The floor collapses 2.3%, about 17 times fewer. The interventional run tells the same story on mean mode mass. Plain RL erodes it. The floor raises it.

Figure 3. Mean change in base-mode log-probability after 150 identical GRPO steps, floor the only difference. Positive means the training rule raised base-mode mass. Only the one-sided floor is positive. UCPO, a reward-shaping coverage method, is the worst arm, below plain GRPO, exactly what the mechanism predicts.

The ordering matters. Global KL genuinely helps, moving collapse from 4.1% to 0.5%, yet it stays net-negative because it applies symmetric pressure toward base. A symmetric base-anchor is stronger and still leaks. UCPO, the purest reward-shaping coverage method, collapses 10% of base modes, worse than doing nothing. A diversity bonus can only act on rollouts it samples, so it never reaches the rare modes and its pressure toward variety among the dominant modes speeds the drift of the rare ones. Only the one-sided off-policy floor reaches zero collapse.

Figure 4. Fraction of base reasoning modes collapsed (more than 10 nats below base) after matched 150-step training. The floor is the only arm at 0%.

Preserved modes turn into coverage

Mode-level preservation shows up as answer-level coverage. On 7B OlympiadBench both floor arms overtake GRPO at every budget from k = 32 up, and the gap widens with k. expPROJ leads at every k ≥ 32 and reaches pass@256 of 0.8566 against GRPO at 0.8497. The cost is a small, bounded pass@1 drop.

Figure 5. pass@k on 7B OlympiadBench, 572 problems. base, plain GRPO, and the two floor implementations. The floor arms trade a little pass@1 for a higher large-k ceiling, and the advantage grows with k.

Coverage is the substrate for the next round

Coverage is not an end in itself. It is the material the next round of RL builds on. We forked round-1 into a plain fork and a floor fork, then ran identical unconstrained round-2 RL on both, and evaluated on the 329-problem Olympiad fragile band at 1024 samples. A second crossover appears, now between the two continued-RL models. The plain fork wins small k from sharpening. The floor fork overtakes at k ≥ 32 and pulls away, reaching 0.990 at pass@256. The round-1 pass@1 tax comes back as a higher round-2 coverage ceiling.

Figure 6. Continued-RL ceiling. After identical round-2 GRPO, the coverage-preserved fork leads at every k ≥ 32 and the gap widens over k = 64 to 256. The plain fork wins small k. The round-1 crossover reproduces between two round-2 models.

Figure 7. The same result at a glance, pass@256 on the fragile band after identical round-2 RL. The coverage-preserved fork reaches 0.990, above the plain fork at 0.983 and the base at 0.976. The axis starts at 0.95 so the gap is visible.

It holds across models and datasets

The effect is not specific to one model or benchmark. Running the frozen method (expSR) against plain GRPO on new model and dataset cells, expSR raises base-mode mass with near-zero collapse while plain GRPO erodes it. The gap tracks how hard plain GRPO would otherwise collapse the base. It is largest on Qwen3-8B, where the plain arm collapses catastrophically at −5.77 nats, and smallest on the more collapse-resistant DeepSeek-Math.

Figure 8. Mean mode-mass change per cell, plain GRPO against expSR, 150 steps, per-cell base-correct bank. expSR preserves where plain GRPO collapses across two model families and two datasets. The Qwen3-8B expSR cell is still running and is omitted here, its plain arm collapses at −5.77 nats.

What is settled and what is not

Settled. A mechanism result bounds an entire family of on-policy methods. Coverage becomes an off-policy feasibility constraint with two working implementations and a direct diagnostic. The coverage gain over GRPO is directionally consistent across model families and datasets, and it scales with how hard plain GRPO collapses the base.

Not yet settled, stated plainly. A statistically significant large-k win is still open. The near-saturated benchmarks leave little headroom, AIME at 7B has no crossover to recover, and the fragile-band gain at k = 1024 is real in sign but small. The theory is stated as propositions with proof sketches. The mechanism predicts the effect grows on harder benchmarks and larger models at k = 1024, so the 14B and 32B grid to k = 1024 is the next experiment.