GRPO: Group Relative Policy Optimisation

Training DeepSeekMath 7B to 51.7% on the MATH competition benchmark without any external tools required something that standard PPO could not deliver cleanly: a policy-gradient update that fits comfortably on the same hardware used for the forward pass. The answer was Group Relative Policy Optimisation (GRPO), introduced by Shao et al. (2024) and subsequently scaled in DeepSeek-R1 to drive the emergent reasoning behaviours that surprised the research community in early 2025.

The core tension GRPO resolves is this: PPO needs a value network (the critic) to compute baselines for variance reduction. For a 7B-parameter policy, training an equally-sized critic alongside it roughly doubles peak memory. GRPO sidesteps this entirely by sampling a group of responses from the current policy for every prompt and computing relative advantages within that group. No separate network; no billion-parameter value head.