Reinforcement Learning Posttraining

machine-learning
posttraining
Train against verifiable rewards first, and measure what happens when the verifier is wrong.

Generation is a sequential decision process: the state is the prompt plus the generated prefix, the action is the next token, and a completed sequence receives a reward. This chapter starts with a finite-action synthetic task, builds verifiable-reward REINFORCE with a group baseline, and treats learned-reward PPO as the secondary path it is at this scale.

Policy gradients

\nabla_\theta J(\theta)= \mathbb{E}\left[ \sum_t\nabla_\theta\log\pi_\theta(a_t\mid s_t) \left(G_t-b(s_t)\right) \right].

The verifiable-reward recipe, in the shape used for math and coding models:

  1. Choose tasks with executable verification: unit tests, format validators, synthetic grammars, arithmetic with known answers.
  2. Sample k completions per prompt.
  3. Score each with the checker and use the group mean as the baseline, A_i = (r_i - \bar{r})/(\operatorname{std}(r)+\epsilon), which removes the need for a learned critic.
  4. Apply the policy-gradient update using the sampled action log-probabilities.

Every component stays inspectable: the reward is a function you can read, and the advantage is arithmetic on k numbers.

Reward loopholes

The required experiment is a deliberately weak checker. Give the verifier a flaw, for example it rewards any completion containing the substring "OK" regardless of the answer, and train against it. At every checkpoint, score the policy with the weak checker (training reward) and with a held-out ground-truth evaluator. Three outcomes are informative:

  • Training reward rises while ground-truth score stays flat: the policy found the flaw. This is reward hacking, and the fix belongs in the checker.
  • Both rise together: the flaw was never reachable at this scale.
  • Ground-truth score falls while reward climbs: full overoptimization; also record generation entropy, since collapse to a single template is the usual mechanism.

Vary group size k (variance of the advantage estimate falls roughly like 1/k), the KL coefficient when a learned reward is in play, and PPO’s clip range once the simpler estimator is understood.

Learned rewards and PPO

When no program can verify the task, a learned reward model from Chapter 09 enters, trained with \mathcal{L}_{\mathrm{RM}} and optimized against through a clipped surrogate,

L^{\mathrm{CLIP}}= \mathbb{E}\left[ \min\left( r_t(\theta)A_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t \right) \right], \qquad r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)},

usually with a KL penalty toward a reference policy, J=\mathbb{E}[r_\phi(x,y)]-\beta D_{\mathrm{KL}}(\pi_\theta\|\pi_{\mathrm{ref}}). Implement this only after the verifiable-reward loop works, because here the evaluator can be exploited and the loophole experiment above is no longer optional.

RL reliability

A policy-gradient update increases the probability of whatever the checker rewards, including completions that satisfy the checker without satisfying the task, whenever the two disagree. The loophole experiment makes that disagreement the measured quantity rather than a surprise: the gap between training reward and ground-truth score at matched checkpoints is the hacking rate, and it is the number to report alongside any claimed RL improvement.

Summary

Verifiable rewards keep the feedback path auditable: reward is a readable function, baseline is group arithmetic, and no learned evaluator sits inside the loop. They do not make the specification correct, which is why the weak-checker experiment and the held-out evaluator are part of the method rather than an appendix to it.

Verifiable-reward implementation

The finite-action derivation is the transparent reference. The shared post-training contract supplies the same group-normalized advantage used by a policy-gradient trainer and a binary reward from the independent proof verifier. A negative or malformed proof receives zero, so the reward path remains executable and auditable.

from proof_lm.logic import generate_examples
from proof_lm.posttraining import group_normalized_advantages, verifier_reward

proof_examples = generate_examples(count=2, seed=10)[:2]
rewards = [verifier_reward(example.proof_text) for example in proof_examples]
advantages = group_normalized_advantages([0.0, 1.0, 1.0, 0.0])
print("verified proof rewards:", rewards)
print("group advantages:", advantages.round(decimals=4).tolist())
print("advantage mean:", round(float(advantages.mean()), 6))
assert rewards == [1.0, 1.0]
assert abs(float(advantages.mean())) < 1e-6
verified proof rewards: [1.0, 1.0]
group advantages: [-1.0, 1.0, 1.0, -1.0]
advantage mean: 0.0

Exercises

Use the exercises to test the chapter’s invariants and connect the derivations to the reusable implementation. Solutions are hidden in the notebook source and are available through the course tooling when needed.

[P10.1] Weak-verifier reward hacking

A verifier rewards a completion whenever it contains the string OK, even when the answer is wrong. Name the failure mode and give one fix.

[P10.2]

Why is a group baseline useful in verifiable-reward policy gradients, and what must be compared to a weak-checker reward to detect hacking?

Back to top