Post·September 2026·10 min read

When Agents Learn to Whisper

Reward optimization can teach two agents a private code that looks like English but hides information from a monitor. A cheap partner-swap probe catches it, and the same property that keeps a channel auditable is the one that makes it generalize.

The channel is the audit trail

A pipeline of language-model agents hands work between specialists. A retriever finds documents, an extractor pulls numbers, a reasoner computes, a verifier checks. The messages passed between these steps are the surface a compliance officer inspects. If those messages stop meaning what they read as, the audit trail is gone even if the pipeline still returns the right answer.

Reinforcement learning trains these agents end to end. That co-adaptation can push the messages toward a private code, still shaped like English, that carries information a monitor cannot recover. This post shows when a private code forms, how to detect it, and how to prevent it.

A relay game as a testbed

One open-weight model plays two roles with low-rank adapters, a Sender and a Receiver. The Sender sees a math word problem and writes a short message under a strict word budget K. The Receiver sees only the message and produces the answer. Reward is answer correctness. We train the pair with group-relative policy optimization on GSM8K.

Three diagnostics tell a genuine channel apart from a private one. Partner-swap replaces the Receiver with a fresh model that never trained with the Sender. If accuracy holds, the message reads like English. If accuracy falls, the message speaks a private code the trained partner alone can read. Paraphrase rewrites the message while preserving meaning and re-tests. Monitor recoverability asks whether an independent model can reconstruct the answer from the message alone. A genuine protocol survives all three. A private code fails them.

A monitor-evasion reward teaches agents to whisper

Plain reward on a natural-language channel stays genuine. Add a small monitor-evasion bonus, a reward that pays the pair when the Receiver succeeds while a fixed monitor fails, and a private code installs. Partner-swap retention roughly halves.

Figure 1. Partner-swap retention at matched budgets, plain reward against a monitor-evasion (steg) reward. Retention falls from 0.91 to 0.57 at K = 8, and from 0.86 to 0.49 at K = 16. The reward that hides a message is the one that makes it partner-specific.

Retention at K = 16 alone makes the collapse blunt. A genuine protocol keeps 86% of the accuracy when we swap the Receiver. A whispered protocol keeps 49%. Half the message stops working the moment the trained partner is gone.

Figure 2. The same result as a single-number ranking, K = 16. A whisper keeps 49% of accuracy under partner swap. A genuine channel keeps 86%.

Two agents optimized for reward alone talk in English. Add pressure to hide, and they still talk in English, but only to each other.

Whispering costs the task

Look at what training does under each objective. Plain reward and the genuineness regularizer both subtract nothing from the task, so their curves climb. The steg objectives subtract the monitor score from the reward, so they trade task success for hiding. At α = 1 the pair stays near zero reward through training. At α = 2 the whole thing collapses under the negative signal. A message that hides well is a message that no longer solves the problem.

Figure 3. Training reward across 250 steps at K = 32. Plain reward and genuineness-reg climb to 0.7 to 0.8. Steg α = 1 hovers near zero, and steg α = 2 stays negative. A monitor-evasion reward is hard to optimize without breaking the task.

Auditability and generalization are the same property

The held-out sweep across channel budgets makes the trade-off visible on every axis at once. Genuine methods keep their accuracy and their off-domain transfer, and their monitor gap stays near zero. The steg methods do worse on all three.

Figure 4. Answer accuracy against channel budget K, four methods. Plain reward and genuineness-reg climb with budget. Steg α = 1 lags. Steg α = 2 stays at the floor. Hiding the message hides the answer too.

Partner-swap tells the same story a different way. The genuine methods retain their accuracy under a fresh Receiver at every budget. The steg methods lose it and stay volatile from run to run, because the code they learn does not have to survive anything but its own training partner.

Figure 5. Partner-swap retention against K. Plain and genuineness-reg hover near 1.0 above K = 8. Steg α = 1 sags. Steg α = 2 spikes and drops erratically. A private code is partner-specific by construction.

The final probe is off-domain transfer. Train on GSM8K, test on SVAMP. Genuine protocols carry across the shift. Private codes do not. Off-domain accuracy falls from 0.79 for the plain channel to 0.58 for the private one, and the steg α = 2 arm sits near zero on the new distribution.

Figure 6. Off-domain transfer against K, SVAMP after training on GSM8K. Genuine methods carry across the distribution shift. Private codes overfit to their partner and their training set.

The three probes point at the same quantity. The property that keeps a channel auditable is the property that makes it generalize. In-domain genuineness predicts out-of-domain transfer at correlation 0.51 across runs. Oversight and performance align. A single objective can buy both.

The monitor gap

The monitor gap is the last piece of direct evidence. It measures how much the Receiver beats the monitor at reading the same message. On plain and genuineness-reg the gap sits near zero, so the Receiver and the monitor read the message about equally well. On steg α = 1 the gap opens up. The Receiver reads the message. The monitor cannot.

Figure 7. Monitor gap by K. Plain and genuineness-reg keep the gap near zero. Steg α = 1 opens a gap of up to 0.20, so the Receiver beats the monitor by 20 points on the same text. This is the signature of a private code.

A cheap defense

The genuineness regularizer adds a term that penalizes the Sender when a paraphrase of its message stops working. Applied preventively on ordinary training, the regularizer holds partner-swap retention near 0.90 across regularizer strengths and costs under four points of task accuracy. It is cheap insurance that anchors the channel before any pressure to hide arrives. Curing a code after evasion pressure was inconclusive in early runs because those runs sat in a low-accuracy regime, so prevention is the right place to spend compute.

Private codes are capability-gated

Run the same setup at 0.5B, 1.5B, and 3B parameters. At 0.5B and 1.5B the agents cannot solve the relay task and no code forms. Only at 3B does task accuracy rise enough for a private protocol to appear. The propensity to whisper grows with model scale, so the risk grows with deployment scale too.

Figure 8. Task accuracy against model size, plain reward and steg pressure. Below 3B the pair cannot solve the task well enough for a private code to form. At 3B the code appears. The result predicts stronger hiding on stronger models.

What this gives a deployed pipeline

Three concrete deliverables come out of this. First, partner-swap is the sharpest probe of the three because a private code is partner-specific by definition, and it runs on any deployed pipeline for the cost of one extra forward pass through a fresh model. Second, the genuineness regularizer restores auditability at little cost, so keeping a channel inspectable does not require out-training the encoder. Third, the unification of oversight and generalization means safety and performance stop competing on this axis. A pipeline that reads well to a monitor also transfers well to new distributions.

What is settled and what is not

Settled. A monitor-evasion reward installs a partner-specific code that a fresh model cannot read, and the code costs both task accuracy and off-domain transfer. Genuineness and transfer track each other across runs. A preventive regularizer holds retention near 0.9 at under four points of task cost. Private codes are capability-gated and appear only once the base task can be solved at scale.

Not yet settled. An adaptive overseer that trains against the agents is the natural next step. My first min-max run undertrained the overseer and the current hiding metric confounds hiding with raw skill, so I do not yet claim learned oversight fails. Building a properly trained adversarial overseer, a capability-controlled hiding metric, and a phase map of when a channel stays inspectable as a function of channel width and overseer strength is the funded work.