DeepSeek-R1 and the Open Reasoning Recipe

When OpenAI shipped o1 in September 2024, the working assumption across the field was that frontier reasoning was a moat - the combination of data, RL infrastructure, and tacit know-how would keep it inside a handful of labs for a generation. In January 2025 DeepSeek released R1: an open-weight model with reasoning capability in the o1 class, a published paper describing the pipeline in workable detail, and six distilled variants between 1.5B and 70B parameters. The moat assumption did not survive the quarter.

R1-Zero: RL from a base model, no SFT

The most surprising result in the paper is R1-Zero: start from DeepSeek-V3 base (a 671B-total / 37B-active MoE), skip supervised fine-tuning entirely, run reinforcement learning with verifiable rewards on maths and code. Reasoning behaviours - long chains of thought, reflection ("wait, let me reconsider"), backtracking, self-verification - emerge from the RL signal alone.

Mechanics:

• Algorithm. GRPO (Group Relative Policy Optimization), a critic-free policy-gradient variant DeepSeek had introduced earlier. Removes the value-model overhead of PPO while keeping its stability.
• Reward. Programmatic - accuracy reward on verifiable answers (maths with known answers, code with unit tests), plus a format reward that the model emits its reasoning between <think> tags. No PRM, no human preference model.
• No SFT. The base model is RL-tuned directly. The "aha moment" graph in the paper shows the average response length climbing dramatically over training - the model is teaching itself to think longer.

R1-Zero hit AIME 2024 pass@1 around 71% (and ~86% with majority voting at 64), competitive with o1-preview, starting from a base model with no reasoning-specific SFT. The failure modes: poor readability (mixed languages, no structure), occasional rambling. Useful for proving the recipe, not for shipping.

R1: cold-start SFT + multi-stage RL

The shipped R1 model adds a small SFT cold-start to fix R1-Zero's readability problems, then runs further RL:

1. Cold-start SFT. A few thousand high-quality long-CoT examples to teach the model a clean reasoning format and reduce the language mixing.
2. RL with verifiable rewards for reasoning, plus a language-consistency reward to discourage mid-chain language switches.
3. Rejection sampling + SFT on the RL-tuned model's own outputs to produce a broader SFT dataset (~800k examples covering reasoning and general tasks).
4. A second RL pass for helpfulness and harmlessness on general prompts.

The result is a model with R1-Zero's reasoning capability and the conversational behaviour expected of a deployed assistant. Reported numbers (from the paper):