Paper·Stanford University·September 2026

Continuous-Utility Direct Preference Optimization

Muhammad Ahmed Mohsin, Muhammad Umer, Emily Fox

Reasoning is not one skill. CU-DPO replaces binary preference labels with continuous utilities over a portfolio of reasoning strategies, recovers the utility-maximizing policy, and lifts strategy-selection accuracy from 35 to 46 percent up to 68 to 78 percent across seven base models.

CU-DPO overview poster — Figure 1. CU-DPO overview. Strategy-conditioned sampling produces K candidate chains. An LLM judge scores each chain with a continuous utility. Progressive refinement lifts low-utility chains into the high-signal band. Two pair sets drive training, strategy selection in Phase 1 and execution refinement in Phase 2.

Reasoning is a portfolio, not one skill

Large language models commit to a single thinking style. They apply the same procedure whether a problem wants a direct computation, a proof, or a case split. Human experts pick the strategy that fits the problem. Models collapse onto one, and that collapse caps their reasoning quality.

Binary preference labels make this worse. Direct preference optimization compares a winner chain and a loser chain with a single bit. A chain with correct reasoning and one late arithmetic slip gets the same reject label as a chain that is wrong from the start. The bit throws away the graded structure we care about.

CU-DPO replaces the bit with a continuous utility. An LLM judge scores each reasoning chain in [0, 1] over correctness, coherence, and step efficiency. Preferences follow Bradley-Terry on the utility gap, so the model learns fine-grained quality instead of a coin flip.

The method

For each problem the base model samples K = 8 chains, one per strategy from a fixed portfolio. Math uses eight strategies, code uses four, causal reasoning uses six. The judge assigns each chain a decomposed utility.

U(x, y) = (1/3) Σ w_c s_c(x, y), with s_c in [0, 1] over correctness, coherence, efficiency.

Bradley-Terry turns utility gaps into preference probabilities, so a larger gap means a stronger preference. Training runs in two phases to keep the signal clean. Phase 1 builds best-vs-rest pairs that fix strategy choice. Phase 2 builds margin-stratified pairs inside a single strategy that fix execution quality. Progressive refinement re-prompts weak chains and keeps a repair only when it raises the utility, which turns partial progress into usable positives.

Separating the two phases matters. One joint phase mixes cross-strategy and intra-strategy pairs, and the gradients fight. Splitting them gives each objective a clean target.

Why continuous supervision recovers the right policy

At the optimum of the DPO objective the learned implicit reward matches the utility up to a problem-dependent constant. The constant cancels inside each problem, so the policy ranks chains by their true utility gaps.

r_θ(x, y) = U(x, y) + c(x). Empirically R² = 0.97 between the learned reward and the judged utility.

Continuous utilities are also cheaper to learn. Observing K utilities directly needs O(NK) samples, while learning the same ranking from binary comparisons needs Ω(NK² log K) . For K = 8 that is a 24x theoretical gain, and about a 2.16x reduction in pairs in practice.

Strategy selection improves across every base model

CU-DPO Phase 1 lifts in-distribution strategy-selection accuracy from the 0.35 to 0.46 range up to 0.68 to 0.78 across seven base models. The model now picks the strategy that fits the problem.

Figure 2. In-distribution strategy-selection accuracy, base model against CU-DPO Phase 1 (Table 6). Every model roughly doubles its accuracy. Top-3 selection and Spearman rank correlation improve in step.

The habit transfers. Trained on DeepMath, HARDMath2, and ProofNet, then tested zero-shot on unseen benchmarks, CU-DPO keeps picking better strategies. Gains grow where a mismatch hurts most.

Figure 3. Out-of-distribution strategy selection (Table 3). Accuracy on GSM8K, MATH-500, and U-MATH after training only on the in-distribution math sets. The largest jump lands on MATH-500 at +23.3 points.

Strategy choice is the primary bottleneck

Step-level supervision fixes execution inside a chain. CU-DPO fixes which chain to run. The two act on different failure modes, so they stack. CU-DPO alone beats Step-DPO, and combining both adds a further gain on every benchmark.

Figure 4. Pass@1 on DeepSeek-R1 8B (Table 2). CU-DPO beats step-level supervision by +3.0 on DeepMath and +4.4 on HARDMath2. The two combine for +7.1 overall, which shows strategy selection and step-level refinement play complementary roles.

Continuous utilities need enough data to pay off

At 25 percent of the data binary DPO holds a small edge, because discrete labels give sharper gradients when samples are scarce. CU-DPO overtakes at 50 percent and the gap widens after that. Rich preference structure needs enough data to calibrate.

Figure 5. Held-out win-rate against training-data fraction on DeepSeek-R1 8B (Table 5). Binary DPO leads at 25 percent. CU-DPO crosses over at 50 percent and pulls ahead through 100 percent.

The utility landscape is real

The gains rest on a measured fact. The best strategy shifts by domain, and no single strategy dominates. Average utility runs high while the gap between best and worst strategy stays wide, so strategy choice carries real signal.

Domain	N	Best strategy	Avg. utility	Avg. margin
Calculus	200	step_by_step (25%)	0.763	0.32
Proof	135	direct (27%)	0.768	0.27
Other	75	step_by_step (19%)	0.819	0.30
Algebra	22	algebraic (27%)	0.755	0.35
Analysis	18	alternative (33%)	0.847	0.28

Table 1. Strategy-problem alignment across 450 training problems. The best strategy changes across domains and the mean best-vs-worst margin is 0.30, which motivates the portfolio.

It generalizes beyond math

The same pipeline runs on code generation and causal reasoning with domain-specific portfolios. On HumanEval CU-DPO reaches 79.2 percent Pass@1, matches Best-of-8 at 1x inference cost, and beats binary DPO by +3.1 points. Qwen3 8B shows a similar +2.9 point gain. The framework needs no architectural change to move across domains.

Takeaway

Treat reasoning as a portfolio and score it with a continuous signal. CU-DPO recovers the utility-maximizing policy, roughly doubles strategy-selection accuracy across seven base models, transfers out of distribution, and stacks with step-level supervision. Binary labels leave this structure on the table.