Applied LLMs
PPO for RLHF in Practice
A concrete walkthrough of how Proximal Policy Optimisation is wired into the RLHF pipeline, covering the four-model setup, the clipped objective, KL penalty shaping, and the failure modes that kill real training runs.
advanced · 8 min read · Premium
OpenAI's InstructGPT demonstrated that a 1.3 B parameter model fine-tuned with RLHF was preferred over the raw 175 B GPT-3 by human evaluators. The mechanism behind that jump is not magic: it is Proximal Policy Optimisation applied to a learned reward signal. Understanding why PPO is used here, rather than simpler gradient methods, and how it fits into the four-model apparatus, is the minimum prerequisite for diagnosing alignment training runs in practice.
The four-model apparatus
RLHF with PPO keeps four distinct models in memory simultaneously. Conflating them is the single most common source of confusion.
Keep reading with Pro.
You're reading the preview. Unlock the full concept plus the library, study plans, the AI mentor, and daily emails.