Cards on the table.

1. Model decides to call a tool and emits {name, arguments} as a structured message.
2. Host executes the tool and returns the result.
3. Model integrates the result and either answers or calls another tool.
4. Repeat until done (capped).

The model never executes anything itself - it only emits intents. The host is the trust boundary.

Underspecified function descriptions. The model picks tools by reading the description field, not the name. "Get user info" tells the router nothing; "Look up a customer's current subscription status by user ID. Returns plan, valid_until, and recent payment history" disambiguates against every other lookup tool. Treat descriptions as prompts to the router, because that is exactly what they are.

• Parallel when calls are independent: fetch customer details and fetch order history simultaneously in one turn.
• Serial when one call depends on another's output: fetch the customer's ID first, then use it to fetch their orders. Serial calls require separate turns because the second call's arguments depend on the first call's result.

Parallel cuts latency dramatically; modern APIs (Anthropic, OpenAI) support multiple tool calls per turn.

1. Iteration cap (3-5 typical) to stop runaway loops.
2. Argument validation against the tool schema before executing - never trust the model's JSON blindly.
3. Sandboxing for any tool that touches the file system, shell, or external network.

The model is a planner, not a safety layer. Treat tool calls as untrusted input.

JSON Schema with three fields: name (machine identifier), description (the natural-language hint the model reads to choose), and parameters (a JSON Schema object defining argument types and which are required). The model returns a tool-call message with the matching name and a JSON arguments payload conforming to the schema. Strict mode on modern APIs guarantees the arguments validate.

• "tool_use" - Claude wants to call a tool. Execute it, append the result to conversation history, and continue the loop.
• "end_turn" - Claude is done. Present the final response to the user. The task is complete.

The agentic loop continues when stop_reason is "tool_use" and terminates when it is "end_turn". This is model-driven termination - the model signals completion, you don't guess.

1. Parsing natural language signals ("I'm done", "Here's the result") - fragile, model phrases things differently each time
2. Arbitrary iteration caps (stop after 5 loops) - cuts off legitimate multi-step reasoning
3. Checking for assistant text content - model can include text alongside tool calls; text does not mean "done"

Always use stop_reason as the loop control mechanism.

After executing a tool, its result must be appended to the conversation history before the next API call. Without this, the model cannot see what happened and will re-request the same tools. Appending results allows the model to:
- Reason about new information
- Decide whether more tools are needed
- Integrate multiple results into a coherent response

A coordinator agent (hub) manages all communication, error handling, and information routing. Specialist subagents (spokes) handle specific domains: search, analysis, synthesis, reporting.

Key rules:
- All communication routes through the coordinator
- Subagents have isolated context (no automatic inheritance)
- The coordinator dynamically selects which subagents to invoke based on query complexity
- Not every query needs the full pipeline

Emit multiple Task tool calls in a single coordinator response rather than across separate turns. This spawns subagents concurrently.

Sequential: Turn 1 spawns Agent A, Turn 2 (after A returns) spawns Agent B. Total time = A + B.

Parallel: Single turn spawns both Agent A and Agent B. Total time = max(A, B).

Overly narrow decomposition. The coordinator breaks a broad topic into too-narrow subtasks that miss entire domains. Example: "impact of AI on creative industries" decomposed into only visual arts subtasks, completely missing music, writing, and film.

The subagents execute correctly within their assigned scope - the problem is always what the coordinator assigned, not how the subagents performed.

The coordinator's allowedTools must include "Task". The Task tool is the mechanism for spawning subagents. Without it, the coordinator cannot delegate work.

Each subagent is defined with an AgentDefinition that includes a description, system prompt, and tool restrictions specific to its role.

Subagents operate with isolated context - they do NOT automatically inherit the coordinator's conversation history or share memory between invocations.

You must include complete findings from prior agents directly in the subagent's prompt. Use structured data formats to separate content from metadata (source URLs, document names, page numbers) so attribution is preserved through the pipeline.

Programmatic enforcement (hooks, prerequisite gates): When errors have financial, safety, or legal consequences. Example: blocking process_refund until get_customer returns a verified ID. Provides 100% compliance.

Prompt-based guidance (system prompt instructions, few-shot examples): When preferences are stylistic or consequences of non-compliance are minor. Provides probabilistic (<100%) compliance.

Rule: if the answer to "what happens if the model skips this step?" is "incorrect refund / data breach / policy violation," use hooks, not prompts.

PostToolUse hooks transform tool results before the model processes them. Common uses:

• Normalize heterogeneous data formats (Unix timestamps to ISO 8601, numeric status codes to human-readable labels)
• Trim verbose tool outputs to only relevant fields (40-field order lookup trimmed to 5 relevant fields)

They are deterministic transformations that happen between tool execution and model reasoning.

When escalating to a human agent, compile a structured handoff summary including:
- Customer ID
- Root cause analysis
- Refund amount / relevant financial details
- Recommended action
- Summary of what was attempted

This is necessary because the human agent receiving the escalation lacks access to the conversation transcript. Everything they need to act must be in the summary.

Good tool descriptions include four elements:
1. Input formats: What identifier types does the tool accept?
2. Example queries: "Use when the user asks about order status"
3. Edge cases: "Returns null if no active orders"
4. Boundaries: "Do NOT use for subscription queries; use get_subscription instead"

Minimal descriptions ("Retrieves customer info") cause unreliable tool selection between similar tools. Tool descriptions are the primary mechanism LLMs use for deciding which tool to call.

1. Transient - Timeout, service unavailable. isRetryable: true.
2. Validation - Invalid input format. isRetryable: false (fix input first).
3. Business - Policy violation (e.g., refund exceeds limit). isRetryable: false. Include customer-friendly explanation.
4. Permission - Insufficient access. isRetryable: false.

Every error must include isError: true, errorCategory, isRetryable, and a human-readable description.

MCP Resources: Expose content catalogs (issue summaries, documentation hierarchies, database schemas). They give agents visibility into available data without requiring exploratory tool calls. Read-only, informational.

MCP Tools: Expose actions (search, create, update, delete). They let agents do things in external systems.

Resources reduce the number of tool calls by showing the agent what data exists before it starts querying.

Key distinction: "any" guarantees a tool call; forced selection guarantees which tool.

4-5 tools per agent is ideal. Giving an agent access to too many tools (e.g., 18) degrades tool selection reliability by increasing decision complexity.

Agents with tools outside their specialization tend to misuse them (a synthesis agent attempting web searches). Restrict each agent to tools relevant to its role, with limited cross-role tools only for specific high-frequency needs.

Project-scoped (.mcp.json): Shared team tooling. Version-controlled. Uses ${ENV_VAR} expansion for credentials. All developers who clone the repo get access.

User-scoped (~/.claude.json): Personal/experimental servers. NOT shared via version control. Only on your machine.

Both are available simultaneously - tools from all configured MCP servers are discovered at connection time.

1. User level (~/.claude/CLAUDE.md): Personal preferences. NOT shared via version control. Only applies to you.
2. Project level (.claude/CLAUDE.md or root CLAUDE.md): Team standards. Shared via VCS. Applies to all team members.
3. Directory level (subdirectory CLAUDE.md): Package-specific rules. Applies to that directory.

Common pitfall: critical conventions in user-level config that new team members don't receive.

Create files in .claude/rules/ with YAML frontmatter containing paths with glob patterns:

paths: ["**/*.test.tsx"]

Rules load only when editing matching files, reducing irrelevant context and token usage. Glob patterns work across directories - ideal for test files, API files, or any convention that spans multiple locations.

Superior to directory-level CLAUDE.md when conventions span multiple directories.

1. context: fork - Run the skill in an isolated sub-agent context. Prevents verbose output from polluting the main conversation. The main session only receives the summary.

2. allowed-tools - Restrict which tools the skill can access during execution (e.g., limit to file reads to prevent destructive actions).

3. argument-hint - Prompt developers for required parameters when they invoke the skill without arguments.

Plan mode: Complex tasks with multiple valid approaches, large-scale changes across many files, architectural decisions (microservice restructuring, library migration affecting 45+ files). Explore first, design, then execute.

Direct execution: Well-scoped changes with clear scope and one obvious approach (single-file bug fix, adding a validation conditional, simple refactoring).

You can combine them: plan mode for investigation, then switch to direct execution for implementation.

The -p (or --print) flag runs Claude Code in non-interactive mode. It processes the prompt, outputs the result to stdout, and exits without waiting for user input.

Without -p, Claude Code waits for interactive input, causing CI jobs to hang indefinitely. This is the documented way to run Claude Code in automated pipelines.

Combine with --output-format json and --json-schema for machine-parseable structured output.

--resume session-name: Continue a prior conversation. Use when prior context is mostly valid and you want to pick up where you left off.

fork_session: Create independent branches from a shared analysis baseline. Use to explore divergent approaches (comparing two refactoring strategies without redoing analysis).

Fresh start with injected summary: New session with a structured summary of prior work. Use when prior tool results are stale (code has changed since last session). More reliable than resuming with outdated context.

"Be conservative" and "only report high-confidence findings" do NOT improve precision. The model interprets these vaguely and inconsistently.

Replace with specific categorical criteria: "Report: bugs, security issues. Skip: minor style, local patterns." Define severity levels with concrete code examples: "Critical = data loss, High = wrong output, Medium = performance regression."

If a specific category has high false positives, temporarily disable it rather than adding vague confidence modifiers.

Few-shot examples are the most effective technique when:
- Detailed instructions alone produce inconsistent results
- You need consistent output format (location, issue, severity, suggested fix)
- Ambiguous scenarios need demonstrated reasoning
- Extraction from varied document structures fails
- You want to reduce false positives

Use 2-4 targeted examples that show reasoning for why one action was chosen over plausible alternatives. Examples enable generalization to novel patterns, not just matching pre-specified cases.

Guarantees: Syntactic validity. The output conforms to the JSON schema - no missing braces, no trailing commas, no type mismatches. Schema syntax errors are eliminated.

Does NOT guarantee: Semantic correctness. The model can produce perfectly formatted JSON with wrong values - line items that don't sum to the stated total, values in the wrong fields, fabricated information for required fields.

For semantic validation, add self-correction patterns: extract calculated_total alongside stated_total, include conflict_detected booleans.

Retries work for: format mismatches, structural output errors, field placement errors. The information exists in the document but the model extracted it wrong. Retry with the original document, the failed extraction, and specific validation errors.

Retries are useless when: the required information is simply absent from the source document. No amount of retrying will extract data that doesn't exist. Mark these as unavailable rather than retrying.

• 50% cost savings compared to real-time API calls
• Processing window: up to 24 hours with no guaranteed latency SLA
• Does NOT support multi-turn tool calling within a single batch request
• Use custom_id to correlate batch request/response pairs
• Good for: overnight reports, weekly audits, nightly test generation
• Bad for: blocking pre-merge checks, real-time code review
• Failure handling: resubmit only failed documents identified by custom_id

A model retains its reasoning context from generation. In the same session, it's less likely to question its own decisions - it remembers why it made each choice and will defend those choices even if they're wrong.

An independent review instance (separate Claude session without the generator's reasoning context) is more effective at catching subtle issues. It evaluates the code fresh, without bias from the generation process.

For large PRs: split into per-file local analysis passes plus a cross-file integration pass to avoid attention dilution and contradictions.

Do you need an agent or a workflow? A workflow orchestrates LLM/tool calls through predefined code paths (you wrote the control flow). An agent lets the model dynamically direct its own process and decide when it is done. Workflows win on predictability for well-defined tasks; agents win when the path genuinely cannot be known in advance. Most "agent" projects are workflows in disguise and would be more reliable built as one.

It provides some subset of: control-flow definition (graph/chain/conversation), persistent state across steps, crash-resume persistence, human-in-the-loop checkpoints, and observability. The cost is an abstraction layer between you and the prompt, which is exactly where agents fail. If you cannot see the exact tokens the model received, you cannot debug it. That trade is the whole decision.

• LangGraph: low-level graph of nodes over shared state; explicit control, persistence, durable long-running agents. Dominates serious production.
• CrewAI: high-level role-based agents that collaborate; fast to start, harder to steer off-metaphor.
• AutoGen: agents as a conversation; elegant for multi-agent research.
• OpenAI Agents SDK: minimal loop with handoffs and guardrails; good for a thin OpenAI-centric stack.

Because the most successful production implementations used simple, composable patterns rather than complex frameworks, adding abstraction only when a simpler version demonstrably failed. A graph engine wrapped around what is really a three-step prompt chain adds failure modes (state bugs, version churn, hidden prompts) without adding capability. Start with plain code and a loop; adopt a framework when its absence is the thing hurting you.

State models and control-flow primitives do not port between frameworks, so switching is a rewrite, not a config change. The graph, the state object, the persistence layer, all are framework-specific. Choose for where you will be in a year, not for the next demo. The corollary: keep the prompt and tool logic separable from the framework so a migration touches plumbing, not reasoning.

It turns the N-times-M integration problem (every application needs custom glue for every tool/data source) into N-plus-M. Build a server once for a data source and a client once in an application, and any compliant client can talk to any compliant server. The official analogy: a USB-C port for AI, one standardised connector instead of bespoke cables per pairing.

1. Tools - model-controlled: functions the model may decide to call (MCP's form of function calling).
2. Resources - application-controlled: data the host loads into context (a file, a row, a document).
3. Prompts - user-controlled: reusable templated workflows a user invokes deliberately.

The "who controls it" axis is the point: tools are the model's, resources are the host's, prompts are the user's.

• Host: the AI application the user interacts with (Claude Desktop, an IDE); runs one or more clients.
• Client: the connector inside the host that maintains a session with a single server.
• Server: a separate process exposing a data source or tool via the protocol.

Because the server runs as its own process, a server written for Slack works unchanged in any MCP-aware host. Transport is stdio (local subprocess) or HTTP (remote), over JSON-RPC.

Tool results flow back into the model's context, so a malicious or compromised server can mount a prompt-injection attack through the data it returns. Treat third-party MCP servers as untrusted input, not trusted plugins. Authorisation is also the protocol's younger part: the tool-calling plumbing standardised faster than auth/permission scoping, so production deployments wrap servers in their own authentication and least-privilege controls.

It standardises how a model reaches a tool; it does nothing for whether the model picks the right tool or uses it well. A bad tool description fails identically over MCP as over a bespoke integration. MCP is a connector, not a brain. It removes integration boilerplate, not the prompt-and-tool-design work that determines whether the agent actually succeeds.

State: the prompt / dialogue context. Action: each token sampled from the vocabulary. Episode: the full response. Reward: a scalar from the reward model, given once at end-of-sequence. The language model itself is the policy being optimised.

Why it matters: understanding this mapping shows why standard PPO applies and why sparse end-of-episode reward makes credit assignment hard.

• \(r_\phi(x, y)\): reward model score for response \(y\) to prompt \(x\).
• \(\pi_\text{ref}\): the frozen SFT-stage reference policy.
• \(\beta\): coefficient trading off reward maximisation against staying close to the reference. \(\beta = 0\) removes the constraint; large \(\beta\) collapses back to imitation.

The KL term prevents reward hacking by penalising drift from the pre-RLHF model.

Annotators compare two responses \((y_1, y_2)\) for the same prompt and label the preferred one. The reward model is trained to maximise the log-likelihood of the preferred response scoring higher:

where \(y_w\) is the "won" (preferred) response and \(y_l\) is the "lost" (rejected) one.

Pairwise comparison is used instead of absolute ratings because humans are far more consistent when making relative judgements.

The reward model \(r_\phi\) is an imperfect proxy for true human preferences. As the policy is optimised harder against \(r_\phi\), it finds inputs that exploit gaps in the proxy: responses that score highly on \(r_\phi\) but that humans would not actually prefer. Gao et al. (2022) measured this as an inverted-U curve: ground-truth performance improves then degrades as KL divergence from \(\pi_\text{ref}\) grows.

This is an instance of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."

Three compounding difficulties:

1. Action space: the vocabulary is roughly 50,000 tokens per step, orders of magnitude larger than most RL benchmarks.
2. Reward sparsity: one scalar is given at the end of a full response (hundreds of tokens), making temporal credit assignment approximate.
3. Pretrained initialisation: gradient steps must be small enough to avoid catastrophic forgetting of the pretraining knowledge, so learning rates and update magnitudes are tightly constrained.

A value network (critic) must generalise across the full context space simultaneously.

1. Annotation noise / bias: human labellers disagree and may prefer long, confident-sounding answers regardless of correctness; the reward model absorbs these biases.
2. Verbosity bias: policies learn to produce longer responses because annotators often rate length as a proxy for helpfulness, even when brevity would be superior.
3. Style collapse: PPO can converge to a narrow, formulaic style that consistently scores well on the reward model, reducing the diversity present in the pretrained model.

All three are expressions of optimising an imperfect proxy rather than the underlying preference.

The sparse terminal reward is distributed across tokens, with the KL term applied at every position:

token_reward[t] = 0.0   # no intermediate reward for most tokens
token_reward[-1] += r_phi(x, y)   # terminal reward at end-of-sequence
# KL penalty at every token:
token_reward[t] -= beta * log(pi_theta(t|ctx) / pi_ref(t|ctx))

PPO treats token_reward as the reward signal. The per-token KL penalty is computed from log-probability ratios between the current policy and the frozen reference, both evaluated on the same generated token. This turns a single end-of-episode scalar into a denser signal while continuously penalising distributional drift.

In a full MDP, intermediate rewards and state transitions across every token would require temporal-difference methods and make credit assignment across thousands of tokens intractable. Treating the complete response as a single "arm" and the prompt as the context collapses this to a one-shot bandit pull: one action (the full response), one scalar reward, no state transitions. This is an approximation, but a practical one that makes the policy gradient update tractable with PPO.

The first term maximises reward from the reward model. The second term penalises the KL divergence from the reference (SFT) policy. The \(\beta\) hyperparameter controls the trade-off: low \(\beta\) allows aggressive reward chasing (risks reward hacking); high \(\beta\) keeps the policy close to the supervised baseline (risks under-alignment). The KL term also preserves linguistic fluency learned during pre-training.

The Bradley-Terry model, fit via a cross-entropy loss on preference pairs:

where \(y_w\) is the preferred response and \(y_l\) the dispreferred one. The assumption is that the probability of preferring \(y_w\) over \(y_l\) depends only on the difference in their scalar rewards. This sidesteps the need for absolute quality scores; annotators only need to rank two options.

Reward over-optimisation occurs when the policy improves its score on the proxy reward model while degrading on the true (gold) reward. Gao, Schulman, and Hilton (2022) measured this empirically and found the relationship is approximately:

The proxy reward climbs monotonically with KL divergence from the reference policy, but the gold reward peaks early and then falls. This is Goodhart's law: the reward model is a noisy proxy, and optimising it too hard reveals exploits the model did not capture. The KL penalty in the objective slows but does not eliminate this effect.

Rafailov et al. (2023) showed that the optimal policy satisfying the KL-regularised RLHF objective is:

Rearranging, the reward can be expressed as a log-ratio of the optimal and reference policies. Substituting into the Bradley-Terry preference loss eliminates the reward model entirely, leaving a loss that trains the policy directly on preference pairs. DPO thus achieves the same objective as PPO-based RLHF without sampling from the policy during training, making it computationally lighter and more stable.

1. Length bias. Annotators systematically prefer longer responses; the reward model encodes length as a quality proxy, and the policy learns to pad outputs.
2. Credit assignment breaks for long-horizon tasks. Multi-step reasoning (coding, maths proofs) requires knowing which intermediate steps were correct. A single end-of-response scalar cannot distinguish sound reasoning that led to a wrong answer from flawed reasoning that accidentally reached the right one. Process reward models re-introduce MDP structure.
3. Out-of-distribution prompts. The reward model is trained on a curated prompt distribution. For prompts outside that distribution the model extrapolates unreliably, and the KL penalty cannot compensate because it has no knowledge of which regions are underrepresented.

Temperature and nucleus (top-p) sampling serve as the exploration mechanism: raising temperature increases the probability mass on lower-ranked tokens, causing the policy to try different response arms. However, coverage remains narrow because the token action space is astronomical (all possible sequences), and which responses are reachable depends heavily on prefix context. The policy explores locally around the current distribution, rarely venturing into genuinely novel regions of response space. This is one reason RLHF training is sensitive to the diversity of the initial SFT demonstrations.

Reward over-optimisation occurs when a policy is trained too aggressively against a proxy reward model, causing it to exploit the proxy's blind spots rather than genuinely improve. It happens because the reward model is an imperfect approximation of human preferences: the policy, as a powerful optimiser, finds inputs the reward model scores highly that do not correspond to truly preferred behaviour. Proxy reward climbs; gold reward peaks and then falls.

• \(r_\phi(x, y)\): the proxy reward model's score for response \(y\) given prompt \(x\).
• \(\beta \cdot \mathrm{KL}[\pi_\theta \| \pi_{\text{ref}}]\): a penalty that increases as the trained policy drifts from the reference (SFT) model's distribution.
• \(\beta\): the coefficient that trades off reward maximisation against staying close to the reference. Larger \(\beta\) reduces exploitation but also limits genuine improvement.

As KL divergence from the reference policy increases (more RL optimisation), proxy reward rises monotonically, but gold reward follows an inverted-U shape: it improves initially, reaches a peak, then declines below baseline. The location and sharpness of the peak depend on reward model size; larger reward models are harder to exploit, so the peak occurs at a higher KL and the decline is more gradual.

1. Length exploitation - the policy pads responses because reward models trained on human data often correlate length with thoroughness.
2. Sycophancy - the policy validates and agrees with the user rather than informing them, because raters tend to prefer responses that confirm their views.
3. Formatting games - the policy inserts bullet points, headers, or other structure regardless of whether it aids comprehension, because reward models associate these with high-quality training examples.

All three emerge without explicit instruction; the policy discovers them as paths of least resistance through the reward model's blind spots.

Ensemble reward models score a response as the minimum (or weighted average) across multiple reward models trained with different initialisations or data subsets. A response must satisfy all models to score well, making it harder to exploit any single model's blind spot.

Weakness: if all ensemble members were trained on the same rater pool and task distribution, they share systematic biases. Behaviours that exploit rater preferences (such as sycophancy) will fool the whole ensemble, because the failure mode is in the data, not the model architecture.

The proxy reward - the signal visible during training - keeps increasing throughout over-optimisation. Training loss curves therefore look healthy; there is no obvious warning sign. True quality has peaked and begun to decline, but this is only visible via periodic human evaluations or a held-out gold reward model that is not used as the training objective. Without such monitoring, a team may continue training past the optimal point while believing the model is still improving.

An outcome reward model scores only the final response, so the policy can reach a plausible-sounding correct answer via flawed or hallucinated intermediate reasoning and still receive high reward. A process reward model assigns a reward signal at each reasoning step, forcing the policy to produce correct intermediate steps as well. This dramatically reduces the routes available for exploitation: the policy cannot simply game the final output format; it must also produce step-by-step reasoning that the PRM validates. The trade-off is that dense step-level labelling is far more expensive to collect than outcome-level comparisons.

An ORM assigns a single scalar reward to the complete response (based only on the final answer). A PRM assigns a reward to each individual reasoning step, providing t signals for a chain of t steps.

Why it matters: dense per-step feedback lets RL penalise intermediate errors even when a lucky recovery produces the right final answer.

The ORM sees only the correct final answer and gives a positive reward, so it effectively reinforces the faulty step-3 reasoning. The PRM scores step 3 directly and assigns it a negative or low reward regardless of the eventual correct answer, discouraging the error.

The policy learns to maximise the proxy reward rather than genuine quality. It finds "shortcuts" (confident-sounding outputs, lucky answer formats, stylistic patterns) that score high on the learned reward model but are not actually correct or helpful. This is reward over-optimisation: the measure becomes the target and stops tracking what we care about.

Process supervision significantly outperformed outcome supervision. Their best process-supervised model reached 78% on a representative MATH test subset. They also released PRM800K: 800,000 step-level human feedback annotations. The PRM improved both average accuracy and consistency (best-of-N reliability improved more than random sampling).

Learned reward models are imperfect proxies: the policy can exploit their blind spots. A verifiable outcome reward (e.g., a unit test, a maths checker) returns ground truth - there is no learned proxy to Goodhart against. The policy can only score a reward by actually producing a correct answer.

Limitation: this only works for tasks where correctness is machine-checkable (maths, code, formal logic), not open-ended generation.

1. PRM reward hacking: the policy learns step phrasings that score high on the PRM without being mathematically valid - the PRM is itself a learned proxy and is gameable.
2. Verbosity exploitation: because each correct step earns reward, the model may pad chains with trivially-true filler steps to accumulate extra signal, inflating chain length without improving accuracy.

PRMs act as step-level scoring heuristics for inference-time search. At each node in a beam search or tree search over candidate reasoning chains, the PRM scores which continuation is most plausible. The best-scoring path is selected or ranked. This decouples the PRM's benefit from the RL training loop entirely, allowing it to improve answer quality at test time without any additional fine-tuning.

Best-of-N generates N independent completions from a fixed policy, scores each with a reward function, and returns the highest-scoring one. It never modifies any model weights -- all the "improvement" comes from spending more inference compute, not from gradient updates.

Each completion is an independent draw from the model's score distribution. The expected maximum of N i.i.d. draws (the N-th order statistic) grows roughly as log N for light-tailed distributions -- sublinear in N. Additionally, coverage saturates once the probability of any sample solving the problem is near 1. Beyond that ceiling, more samples add nothing because the model's latent capability is fully exercised.

An ORM scores only the final completed sequence; it is blind to intermediate reasoning steps, so a completion with flawed reasoning that reaches a correct answer looks the same as a well-reasoned one. A PRM scores each reasoning step and can penalise a completion that has a weak or incorrect intermediate step, making it harder to game with answer-hacking. PRMs are more expensive to label and train but more robust at high N.

At large N, you are exhaustively searching for the completion that maximises the reward model's score. The reward model is a proxy for true quality. Eventually, the argmax winner is the completion best adapted to the reward model's blind spots or biases -- not the genuinely best answer. This is reward overoptimisation at inference time: the metric (reward model score) ceases to track the true goal once it becomes the selection target.

Coverage is the fraction of problems for which at least one of the N samples is correct. It upper-bounds what any selection strategy can achieve: if the correct answer never appears in N samples, no verifier can surface it. Coverage scales log-linearly with N (Brown et al., 2024) when an exact verifier is available. Average performance (expected reward) can be misleading because a single correct sample in N is enough to get credit; coverage makes this threshold explicit.

Best-of-N needs diverse samples to cover the space of possible answers. At temperature near zero, all N completions collapse toward the greedy decode -- you get N nearly identical outputs, no coverage gain, and wasted compute. At high temperature, individual completions become incoherent or factually unreliable, so coverage of correct answers may actually fall. The useful operating point is somewhere in between, and it is task-specific; calibrating temperature is a real practical concern.

Snell et al. showed that allocating compute uniformly (same N for every prompt) is inefficient. A compute-optimal strategy -- giving more samples to harder prompts estimated via a difficulty predictor, and fewer to easy ones -- achieves more than 4x efficiency improvement at the same total compute budget compared to uniform Best-of-N. They also found that a small model with optimal test-time compute can outperform a 14x larger model running a single greedy decode on prompts in the right difficulty range.

RFT samples multiple completions per prompt from the current model, discards those that are incorrect (reward = 0), and trains on the survivors via supervised cross-entropy. PPO instead computes a gradient from every rollout (weighting by advantage), uses an explicit clipped surrogate objective to control step size, and requires a separate critic network to estimate baselines. RFT is structurally simpler: no critic, no clipping, and no gradient from incorrect samples.

The RFT loss on accepted completions Y+ is:

L_RFT = - E_x [ (1/|Y+|) * sum_{y in Y+} log pi_theta(y | x) ]

Taking the gradient and recognising that Y+ contains exactly the samples where r(y|x) = 1:

grad L_RFT ≈ - E[ r(y|x) * grad log pi_theta(y | x) ]

This is the REINFORCE estimator with binary rewards and no explicit baseline. The zero-reward samples contribute nothing because they are not included in Y+; their implicit gradient weight is zero. RFT is REINFORCE in disguise.

Iterative RFT repeatedly cycles through: (1) sample completions from the current policy, (2) filter to correct ones, (3) fine-tune on survivors, (4) update the policy and repeat. Because each round's training data is generated from the policy as it currently stands, the data is on-policy. This satisfies the core requirement of policy gradient methods and is structurally identical to a simplified PPO loop without clipping or a critic. The Llama 2 post-training alternated this procedure with PPO across training stages.

Best-of-N generates N completions at inference time, scores each with a reward model or verifier, and returns the top-scoring one. No gradient flows; it is a pure selection procedure. Gao et al. (2022) showed that as N scales, proxy reward grows roughly as O(log N) but gold reward follows an inverted-U: it peaks and then declines. This same over-optimisation curve appears in PPO and in iterative RFT. The shared mechanism is any optimisation procedure (gradient-based or sampling-based) searching under an imperfect reward signal. RFT is not immune simply because it avoids an explicit RL loop.

When none of the G sampled completions is correct, Y+ is empty. No samples survive the filter, so the cross-entropy loss has no terms for this prompt and no gradient flows. The model receives no update signal, however hard or near-miss the attempts were. GRPO avoids this by computing within-group relative advantages: when all rewards are equal (all zeros), all advantages are zero and the gradient is near-zero but still defined. RFT's hard filter makes training blind to hard problems where the model's pass rate approaches zero.

RFT has no explicit KL regularisation term. In practice three informal substitutes are used: short training duration (few epochs per round), early stopping when validation reward plateaus, and sometimes mixing filtered data with original SFT data as a soft anchor. When none of these safeguards is applied and iterations run aggressively, the policy drifts far from the reference without any corrective force. The result is distribution collapse: the model converges to a narrow set of solution templates that matched the filter on training prompts but generalises poorly, the same pathology as over-optimised PPO but with no built-in control mechanism.

1. Sample inefficiency on hard prompts. When the correct-sample rate is low (say 1 in 64), the majority of rollout compute is discarded. PPO extracts a gradient from every rollout by weighting by advantage; RFT throws away all incorrect samples. At 70B parameters, generating 64 rollouts per prompt per iteration is expensive, so this waste is directly proportional to inference cost.

2. No signal on zero-pass-rate prompts. As noted above, prompts where the model never succeeds produce no training data. This creates a systematic blind spot for the hardest problems in the distribution, causing iterative RFT to specialise on problems the model can already partially solve rather than expanding capability at the frontier.

Singhal et al. (2023) showed that a policy trained with a reward based solely on response length reproduced most of the downstream win-rate gains achieved by a full RLHF policy. This implies that a significant fraction of apparent quality improvement from RLHF may reflect length exploitation rather than genuine reasoning improvement.

β scales the KL penalty KL(π || π_ref) that keeps the trained policy close to the reference. A low β lets the policy drift far enough to discover and exploit proxy reward weaknesses (length, formatting). A high β prevents that drift but also limits genuine learning. Reward hacking lives in the low-β regime where the policy has enough freedom to find surface-level shortcuts.

1. Spurious headers ("## Background", "## Answer") - mimic expert document structure and signal organisation.
2. Bullet decomposition of a single idea into multiple points - looks more thorough and effortful.
3. Redundant conclusions that repeat the answer - triggers annotator preference for perceived completeness.
4. Confidence theatre ("I am confident that...") - matches annotator preference for assertive, authoritative tone.

All four are cheap to produce and reliably raise reward without improving content quality.

LLM judges inherit the same length and formatting biases present in their training data. Dubois et al. (2024) showed that AlpacaEval's GPT-4 judge assigns higher win rates to longer responses independently of quality. In RLAIF, the policy is optimised against this biased judge, so it learns to produce long and heavily formatted responses. Human evals of the resulting policy then show win-rate gains that are partly or wholly attributable to length rather than reasoning. The bias compounds across iterations.

The curve plots ground-truth quality against the KL divergence from the reference policy. It rises initially (the proxy reward and true quality agree), reaches an interior maximum, then falls as the policy exploits proxy weaknesses. The implication: there is an optimal stopping point for RL training. Training longer against the proxy reward degrades true quality even as proxy reward continues to rise. This is Goodhart's Law in action: once a measure becomes a target, it ceases to be a good measure.

RLVR uses a binary reward on the final answer (correct/incorrect), which removes direct length incentives on the answer token. However, the chain-of-thought reasoning trace is unconstrained. If longer reasoning traces correlate with correct answers in training data (even spuriously), the policy can learn to pad or repeat reasoning steps. Format hacking in the trace (e.g., re-stating the problem, redundant recaps) also emerges because the binary reward signal does not penalise it. The hacking shifts from the output to the reasoning scaffold.

• Length penalty r_adj = r_φ - λ·|y|: directly reduces verbosity but risks penalising legitimately detailed answers; λ calibration is hard.
• Format-invariant reward (strip markdown before scoring): removes surface cues but may remove genuine structural signals.
• Length-controlled evaluation (AlpacaEval 2.0 LC regression): debiases metrics but does not fix training gradients.
• Diverse preference data with short-preferred pairs: reduces the reward model's length prior but requires annotator discipline.
• Higher β (KL coefficient): limits policy drift and exploitation depth but slows and caps learning.

No single fix is complete; practitioners typically combine two or more.

S - state space (all situations the agent can be in); A - action space (all choices available); P(s'|s,a) - transition function (probability of reaching s' from s via a); R(s,a,s') - reward function (scalar feedback per transition); gamma - discount factor (how much future rewards are down-weighted).

Together these five elements fully specify the decision problem an RL agent must solve.

The Markov property states:

P(s_{t+1} | s_t, a_t, s_{t-1}, a_{t-1}, ...) = P(s_{t+1} | s_t, a_t)

The next state depends only on the current state and action, not on the full history. Without this assumption the state space would grow exponentially with time (every distinct history would be a different "state"), making exact solution methods like dynamic programming computationally infeasible.

V^pi(s) = sum_{a} pi(a|s) * sum_{s'} P(s'|s,a) [ R(s,a,s') + gamma * V^pi(s') ]

• pi(a|s) - probability of choosing action a under policy pi
• P(s'|s,a) - transition probability to successor state s'
• R(s,a,s') - immediate reward for that transition
• gamma * V^pi(s') - discounted value of the successor state

The equation is recursive: the value of a state is defined in terms of the values of its successors. This recursion is what dynamic programming and TD learning methods exploit.

The expectation equation averages over the current policy pi(a|s):

V^pi(s) = sum_a pi(a|s) * sum_{s'} P(s'|s,a) [R + gamma * V^pi(s')]

The optimality equation takes the maximum over actions instead:

V*(s) = max_a sum_{s'} P(s'|s,a) [R + gamma * V*(s')]

The optimality version finds the best possible policy; it no longer depends on a fixed pi. Once V is known, the optimal policy is simply "act greedily with respect to V".

• V^pi(s): expected return starting in state s and following policy pi from the very next step.
• Q^pi(s, a): expected return if you take action a in state s (regardless of what pi says), then follow pi afterwards.

Relationship:

V^pi(s) = sum_a pi(a|s) * Q^pi(s, a)

Q is more useful for control because comparing Q(s, a) values across actions lets an agent improve its policy without needing the transition model P.

1. Partial observability: the agent cannot see the full state (e.g., hidden opponent cards). The correct model is a POMDP; exact solutions are PSPACE-hard. Workaround: recurrent networks or belief-state estimation.

2. High-dimensional / continuous state spaces: exact dynamic programming is intractable (the "curse of dimensionality"). Workaround: function approximation (neural networks), but convergence guarantees largely disappear.

3. Reward misspecification / hacking: the MDP assumes R is given correctly. In practice, a poorly designed reward leads to unintended high-return policies (e.g., a robot that covers its camera to avoid seeing mess). Workaround: reward modelling, human feedback (RLHF).

gamma down-weights future rewards: a reward k steps away is worth gamma^k today.

• gamma = 0: agent is fully myopic - it only maximises the immediate next reward and ignores all future consequences. Useful for toy analysis but rarely correct in practice.
• gamma -> 1: the agent values the distant future almost as much as the present. The infinite-horizon return may diverge unless episodic termination is guaranteed. Even when it converges, high gamma inflates return variance, making learning slow and unstable.

In practice gamma is a hyperparameter (commonly 0.95 to 0.999); there is no principled formula to derive it from the task structure.

The reward \(r_t\) is a scalar signal for one transition. The return \(G_t\) is the cumulative sum of all future rewards from time \(t\) onwards:

The agent optimises the expected return, not individual rewards. A high single reward that leads to a long sequence of losses is worse than a modest reward that starts a profitable trajectory.

Converges when \(\gamma \in [0,1)\) and rewards are bounded. The geometric series \(\sum \gamma^k = \frac{1}{1-\gamma}\) provides the upper bound.

Without the condition \(\gamma < 1\), the sum can diverge for a continuing task with nonzero rewards, making it useless as an optimisation target.

The effective horizon is approximately \(\frac{1}{1-\gamma}\) steps. Beyond this point, the discounted contribution of a reward falls below \(e^{-1} \approx 0.37\) of its face value.

Choosing \(\gamma\) is a bias-variance tradeoff: lower \(\gamma\) reduces variance but biases the agent towards myopia; higher \(\gamma\) captures long-range structure but makes gradient estimates noisier.

• Terminated: the agent reached a true terminal state (e.g., game over, goal achieved). The bootstrap value is 0.
• Truncated: the episode was cut short by a time limit, not by the environment. The future continues; the bootstrap value should be the estimated value of the final state.

If you treat truncation as termination (bootstrap = 0), every return in that trajectory is biased downward. Modern gym-style APIs expose a separate truncated flag to prevent this.

In an episodic task the sum is always finite because the episode terminates at time \(T\). Setting \(\gamma = 1\) just means all future rewards are weighted equally, which is valid.

In a continuing task there is no \(T\), so \(\sum_{k=0}^{\infty} 1 \cdot r_{t+k+1}\) may diverge if rewards are nonzero. The solution is either to enforce \(\gamma < 1\) or to switch to the average-reward formulation, which subtracts the long-run mean reward at each step.

Because \(G_t = r_{t+1} + \gamma G_{t+1}\), each return depends on the next one. The backward pass exploits this recurrence in \(O(T)\) time:

G = last_value   # 0 if terminated, V(s_T) if truncated
for r in reversed(rewards):
    G = r + gamma * G

last_value = 0 for true termination. last_value = V(s_T) for truncation, bootstrapping from the value estimate so the agent is not penalised for the environment cutting the episode short.

1. Myopia from low \(\gamma\): The agent ignores rewards more than a few steps away, sacrificing long-run goals for immediate scraps (e.g., collecting a small reward in the wrong corridor instead of walking to the goal).

2. Terminal state misidentification: Forgetting to zero the bootstrap on true termination inflates value estimates near terminal states, because the function learns to predict rewards that will never occur. This is a silent bug with no obvious error signal during training.

V(s) is the expected cumulative return from state s under a given policy. Q(s, a) additionally conditions on the first action taken, so Q(s, a) = E[G_t | S_t=s, A_t=a]. Q is preferred when you need to select actions without a model of the environment (Q-learning, DQN), because the greedy action is simply argmax_a Q(s, a). V is preferred as a baseline or critic in actor-critic methods where a separate policy network handles action selection.

Q^pi(s, a) = sum_{s', r} p(s', r | s, a) * [ r + gamma * sum_{a'} pi(a'|s') * Q^pi(s', a') ]

• p(s', r | s, a): transition and reward distribution (environment dynamics).
• r: immediate reward on this step.
• gamma: discount factor, down-weighting future rewards.
• sum_{a'} pi(a'|s') Q^pi(s', a'): expected value of the next state under the current policy, expressed as a policy-weighted sum over Q values.

The equation says: the value of taking action a in state s equals immediate reward plus discounted expected next-state value. It is a self-consistency constraint; any Q satisfying it for all (s, a) is the true Q^pi.

The expectation equation averages over the policy: sum_{a'} pi(a'|s') Q^pi(s', a'). The optimality equation replaces the average with a max: max_{a'} Q*(s', a'). This means Q is the value achievable by the best possible policy from the next state onward, not the current policy. The consequence: once Q is found, the optimal policy is greedily derived as pi*(s) = argmax_a Q*(s, a), with no need to store an explicit policy. Q-learning targets Q* directly (off-policy), which is why it converges to the optimal policy regardless of the exploration strategy used.

The TD error (delta) is:

delta = r + gamma * V(s') - V(s)

It measures how much the current estimate V(s) disagrees with a one-step bootstrapped target (r + gamma * V(s')). A positive delta means the state turned out better than expected; a negative delta means worse. The TD(0) update rule V(s) += alpha * delta nudges V(s) toward the target by a step of size alpha. Over many samples, this drives V toward V^pi by iteratively reducing the Bellman residual.

The deadly triad is the combination of: (1) function approximation (e.g., a neural network), (2) bootstrapping (using estimated values as targets, as in TD learning), and (3) off-policy data (training on transitions not generated by the current policy). Each alone is manageable. Together, the bootstrap target depends on approximated values, the approximation error feeds back into future targets, and the off-policy distribution mismatch means gradient updates may move parameters in directions that worsen estimates on frequently-visited states. The result can be divergence rather than convergence. DQN partially mitigates this with a lagged target network and experience replay, but does not eliminate the theoretical risk.

The Bellman operator is a contraction with factor gamma: each application brings value estimates gamma-times closer to the fixed point. When gamma is near 1, this contraction is weak (the factor approaches 1), so many iterations are needed for convergence, and estimation noise at distant time steps compounds heavily. When gamma is low, convergence is fast but the agent is myopic and ignores long-horizon consequences. There is no universally correct gamma; it must be tuned to the task's effective horizon. In sparse-reward environments with long episodes, a high gamma is necessary but makes training fragile.

Define the Bellman operator T^pi acting on a value function V:

(T^pi V)(s) = sum_a pi(a|s) sum_{s',r} p(s',r|s,a) [r + gamma * V(s')]

V^pi is a fixed point: T^pi(V^pi) = V^pi. Because T^pi is a contraction (by factor gamma < 1) under the sup-norm, Banach's fixed-point theorem guarantees a unique fixed point and that repeated application converges from any starting point. For algorithms this means: tabular policy evaluation (repeatedly applying T^pi to any initial V) is guaranteed to converge to V^pi. TD learning is a stochastic approximation of the same iteration. With function approximation the contraction no longer holds in general, which is where the deadly triad problems enter.

Value-based methods parameterise a value function (V(s) or Q(s,a)) and derive a policy implicitly by acting greedily with respect to it. Policy-based methods parameterise a policy π_θ(a|s) directly and optimise expected return via gradient ascent on that policy. One represents the value landscape; the other represents the behaviour directly.

DQN selects actions via argmax_a Q(s, a). When the action space is continuous this argmax is intractable - you cannot enumerate all possible actions. Discretising loses resolution; solving a continuous optimisation at each step is expensive. DDPG works around this by adding an actor network that learns to approximate argmax_a Q(s, a), effectively grafting a policy method on top of a value method.

The policy gradient estimator ∇_θ log π_θ(a|s) · G_t is only unbiased when the trajectory (s, a) was sampled from the current policy π_θ. Reusing off-policy data introduces bias unless importance-sampling corrections are applied, and those corrections increase variance rapidly as the behaviour policy diverges. The practical cost is data inefficiency: each collected batch can only be used for a small number of gradient steps before it must be discarded.

The critic estimates a baseline or advantage A(s, a) = Q(s, a) - V(s). This replaces the high-variance Monte Carlo return G_t in the gradient estimator with a lower-variance bootstrapped estimate. The actor still performs policy gradient updates, but the signal it receives is far less noisy, allowing stable learning with shorter rollouts. Without the critic, sparse or delayed rewards produce extremely noisy gradients.

The deadly triad (Sutton & Barto) refers to the combination of: (1) function approximation, (2) bootstrapping (using estimated values as regression targets), and (3) off-policy training. When all three coincide, Q-value estimates can diverge. Value-based methods are most exposed because they routinely use all three - neural network approximators, Bellman bootstrap targets, and replay buffers with off-policy data. Policy gradient methods are less exposed because they update via on-policy returns, not bootstrapped value targets.

A stochastic policy is strictly necessary when the optimal strategy is a mixed strategy - for example, in a two-player zero-sum game where a deterministic policy is exploitable, or in a partially observed environment where multiple actions have the same expected value given the observable state. Policy-based methods naturally represent stochastic policies because π_θ(a|s) is an explicit probability distribution. Value-based methods are inherently deterministic (they output an argmax), so they cannot represent mixed strategies without additional mechanism.

PPO constrains how far the updated policy can move from the behaviour policy that collected the data by clipping the probability ratio r_t(θ) = π_θ(a|s) / π_θ_old(a|s) into the interval [1-ε, 1+ε]. This limits the importance-sampling error introduced by reusing data across multiple gradient steps. It does not eliminate the on-policy requirement entirely - data is still discarded after a small number of epochs - but the clip makes several passes over each batch safe enough in practice without a full importance-weighted correction.

You need (1) a complete transition model P(s' | s, a) and (2) a reward function R(s, a, s'). Without these, you cannot perform the expectation over successor states that the Bellman update requires. This is why DP methods are called "model-based" - they assume the MDP is fully known.

The optimal value of state s equals the maximum over all actions of the expected discounted return:

V*(s) = max_a  sum_s'  P(s'|s,a) [R(s,a,s') + gamma * V*(s')]

In words: "choose the action whose expected immediate reward plus discounted optimal future value is highest." The max replaces the policy-weighted sum in the expectation equation, encoding the greedy improvement step.

The policy improvement theorem states that the greedy policy pi' derived from V^pi satisfies V^{pi'}(s) >= V^{pi}(s) for all states s, with strict improvement unless pi was already optimal. Because there are finitely many deterministic policies (|A|^|S|), and each iteration produces a strictly better or unchanged policy, policy iteration must terminate at the optimal policy in a finite number of steps.

For any two value functions V and V', the Bellman optimality operator T satisfies:

||TV - TV'||_inf  <=  gamma * ||V - V'||_inf

Because gamma < 1, T is a contraction mapping under the sup-norm. By Banach's fixed-point theorem, repeated application of a contraction converges geometrically to a unique fixed point (V*). Each sweep reduces the error by at least a factor of gamma, so value iteration converges, though only asymptotically - you stop at an epsilon-optimal solution.

Policy iteration runs policy evaluation to full convergence (iterating Bellman expectation updates until the value function stabilises) before taking a single greedy improvement step. Value iteration collapses this into one operation: it applies a single Bellman optimality update per state per sweep, which simultaneously evaluates and improves. Policy iteration uses fewer outer iterations but each is more expensive; value iteration has cheaper iterations but needs more of them to converge.

1. No transition model available. Real environments (robotics, games, finance) do not provide P(s'|s,a) analytically. DP is inapplicable without it, which drove model-free methods that learn from sampled experience instead.

2. State-space explosion. DP sweeps cost O(|S|^2 * |A|) per iteration. Even modest real-world problems have state spaces far too large for tabular representation - an Atari screen has on the order of 10^70 possible pixel configurations. Neural function approximation (e.g. DQN) sidesteps tabular storage at the cost of losing the convergence guarantees DP provides.

Generalised policy iteration (GPI) is the observation that almost every RL algorithm can be understood as some interleaving of (1) policy evaluation - updating a value function estimate towards consistency with a policy - and (2) policy improvement - updating the policy to be greedier with respect to the current value function. The two processes interact and compete but jointly converge to the optimal value function and policy.

Policy iteration: full evaluation before each improvement. Value iteration: one-step evaluation before each improvement. TD learning and Q-learning: online, sampled, approximate evaluation with continuous greedy improvement. All sit on the same GPI spectrum, differing only in how deeply they evaluate before improving and whether they use a model or samples.

It averages the actual discounted returns G_t obtained from complete episodes that passed through state s. The estimate is unbiased because no bootstrapping occurs: each G_t is the true realised return under the current policy, not an approximation constructed from other estimates.

First-visit MC uses only the return from the first time state s is visited per episode; every-visit uses the return from every visit. Both converge to V_π(s), but first-visit produces independent samples within an episode, giving it cleaner theoretical properties (the samples are i.i.d. across episodes).

G_t is the sum of all future rewards along a trajectory, accumulating randomness from every stochastic action and transition until episode end. TD(0) uses only a single step R_{t+1} + γV(S_{t+1}), so its updates see far less compounded randomness. Bias and variance trade off: removing bootstrap bias (MC) costs you variance.

A deterministic policy will never visit state-action pairs outside its chosen actions, leaving those Q-values uninitialised. Without coverage of all pairs, greedy improvement may lock onto a suboptimal policy. Exploring starts or epsilon-greedy exploration guarantees every (s, a) pair is visited infinitely often, which is necessary for convergence.

The ratio corrects for the mismatch between behaviour policy b and target policy π:

ρ = Π_{t} [π(A_t|S_t) / b(A_t|S_t)]

Each factor is a probability ratio. For a T-step episode the product of T independent ratios has variance that grows exponentially with T. When π and b differ substantially, many ratios will be far from 1 and the weighted return estimate becomes highly unstable.

TD(λ) targets are a geometric mixture of n-step returns:

G_t^λ = (1-λ) Σ_{n=1}^∞ λ^{n-1} G_t^(n)

Setting λ = 0 recovers TD(0) (one-step bootstrap). Setting λ = 1 recovers the full Monte Carlo return (provided the task is episodic). Intermediate values interpolate the bias-variance trade-off continuously, which is why GAE uses a tunable λ for policy gradient variance reduction.

1. Continuing tasks with no terminal state: MC needs an episode to end before updating, so it simply cannot run.
2. Very long episodes: the variance of G_t grows with episode length, making sample efficiency prohibitively poor.

TD methods handle both cases cleanly because they update after every single step without waiting for episode termination.

V(s_t) ← V(s_t) + α [r_{t+1} + γ V(s_{t+1}) - V(s_t)]

• α: learning rate
• r_{t+1} + γ V(s_{t+1}): the TD target (one-step lookahead)
• r_{t+1} + γ V(s_{t+1}) - V(s_t): the TD error δ_t (prediction error to correct)

Why it matters: TD learns online after every step, not after every episode, because the target substitutes V(s_{t+1}) for the unknown true return.

Q-learning target: r + γ max_a Q(s', a) - uses the greedy action regardless of what the agent actually does next.

SARSA target: r + γ Q(s', a') - uses the actual next action a' drawn from the current policy.

Q-learning is off-policy because the target is independent of the behaviour policy. SARSA is on-policy because its target reflects the exploration the agent will actually perform. In dangerous environments, SARSA is more conservative because it accounts for exploratory stumbles; Q-learning can be overoptimistic.

The deadly triad is the combination of:
1. Off-policy learning
2. Bootstrapping (using estimated future values in the target)
3. Function approximation (e.g., a neural network)

Any two of these are individually fine. All three together can cause value estimates to diverge rather than converge, even when learning rate and architecture are sensible. DQN mitigates the problem with experience replay and a separate frozen target network, but does not eliminate the theoretical risk. Tabular Q-learning is safe because it has no function approximation.

G_t^(n) = r_{t+1} + γ r_{t+2} + ... + γ^{n-1} r_{t+n} + γ^n V(s_{t+n})

• n=1 → TD(0): one real reward, then bootstrap.
• n=∞ → Monte Carlo: all real rewards, no bootstrap.

Intermediate n reduces the variance of MC (fewer summed noise terms) while reducing the bias of TD(0) (less reliance on the current, possibly inaccurate V). In practice, values around n=3-10 often beat both extremes.

In stochastic environments, Q(s', a) estimates have noise. max_a Q(s', a) is always >= the true max value in expectation - you systematically select the positively-biased noise outlier. This inflates targets and propagates overestimates.

Double DQN decouples two steps:
1. Action selection: use the online network to pick arg max_a Q(s', a).
2. Action evaluation: use the frozen target network to compute Q_target(s', selected_a).

Because selection and evaluation use independent (or semi-independent) networks, the positive correlation that drives maximisation bias is broken. van Hasselt et al. (2015) showed this reduces overestimation and improves final performance on Atari.

δ_t = r_{t+1} + γ V(s_{t+1}) - V(s_t)

It is the signed difference between what was predicted (V(s_t)) and what the evidence now suggests the value should be (r_{t+1} + γ V(s_{t+1})).

Neuroscientists have noted that dopamine neuron firing in the brain resembles TD error signals: dopamine neurons fire at unexpected reward, are suppressed when predicted reward is omitted, and shift their response from reward to the reward-predicting cue as learning proceeds - exactly the behaviour TD errors drive. Schultz, Dayan & Montague (1997) formalised this connection.

TD(0) propagates value information one step per visit. In sparse-reward environments (e.g., a maze where +1 appears only at the exit), value estimates near the start state remain near zero until states adjacent to the goal have been visited many times and their values have propagated backward step by step.

Techniques that accelerate credit assignment:
- Larger n in n-step TD or TD(λ): cover more ground per update.
- Reward shaping: add intermediate pseudo-rewards that guide the agent.
- Hindsight Experience Replay (HER): relabel failed trajectories as having achieved the state they actually reached, generating useful signal from every transition.
- Priority-based replay (Prioritised Experience Replay): sample transitions with high δ_t more often, focusing updates where the estimates are most wrong.

Q(s, a) ← Q(s, a) + α · [r + γ · max_a' Q(s', a') − Q(s, a)]

• α: learning rate (step size)
• r: immediate reward observed after taking action a in state s
• γ: discount factor, down-weighting future rewards
• max_a' Q(s', a'): bootstrapped estimate of optimal future value from next state s'
• [...]: the TD error - the surprise signal driving the update

Why it works: repeated application of this contraction operator drives Q towards Q*, the true optimal action-value function.

Q-learning targets r + γ · max_a' Q(s', a'), which assumes the greedy action will be taken next, regardless of what action the behaviour policy (e.g. ε-greedy) actually selects.

The policy being evaluated and improved (greedy) differs from the policy generating experience (ε-greedy). That separation is what makes it off-policy.

Practical payoff: experience from old policies can be reused in a replay buffer, because the target is not tied to how that experience was collected.

When Q(s', a') values have estimation noise, max_a' Q(s', a') systematically picks the overestimated action. This positive bias accumulates through bootstrapping, inflating Q values beyond their true optimum - especially in stochastic environments.

Double Q-learning fix: decouple action selection from action evaluation using two separate estimators θ and θ'. One network selects the greedy action; the other evaluates it:

target = r + γ · Q(s', argmax_a' Q(s', a'; θ); θ')

The overestimation bias is substantially reduced because the two networks' errors are independent.

1. Experience replay: stores transitions in a circular buffer; mini-batches are sampled uniformly at training time. Breaks the temporal correlations between consecutive observations that destabilise stochastic gradient descent.

2. Target network: a periodically-copied frozen version of the Q-network generates bootstrap targets. Prevents the "moving target" problem where the network chases its own shifting predictions and diverges.

Both address the same root cause: neural networks trained with bootstrapped, auto-correlated targets are prone to oscillation and divergence.

The deadly triad (Sutton & Barto): function approximation + bootstrapping + off-policy learning.

Each alone is manageable:
- Supervised learning uses function approximation but not bootstrapping.
- Tabular Q-learning uses bootstrapping but not generalising approximators.

Together they can cause divergence: the approximator generalises errors across states, bootstrapping feeds those errors back as targets, and off-policy data violates the distribution assumptions that keep the approximator stable. DQN operates squarely in this regime - it works empirically but has no convergence guarantee.

The update and the greedy policy both require computing max_a Q(s, a) over the entire action set. In a discrete action space with N actions this costs O(N) forward passes. In a continuous action space the argmax is an optimisation problem with no closed form for a general Q-network.

Workarounds:
- Discretise the action space (loses precision, scales poorly).
- Use a special network architecture where the argmax is analytic (e.g. NAF - Normalised Advantage Functions).
- Switch to policy gradient methods (DDPG, SAC, PPO) that parameterise a policy directly and never need an explicit Q-maximisation step.

Watkins & Dayan (1992) proved convergence requires:

1. All (s, a) pairs visited infinitely often - the agent must explore sufficiently (e.g. ε-greedy with ε > 0 forever, or decaying ε that sums to infinity in visits).
2. Robbins-Monro learning rate schedule: Σ_t α_t = ∞ and Σ_t α_t² < ∞ - the schedule must step far enough to correct any initialisation, but shrink fast enough that noise averages out.
3. Bounded rewards and a finite MDP.

In practice α is often fixed (violating condition 2), which means Q converges to a neighbourhood of Q rather than Q exactly; this is acceptable in most empirical settings.

Experience replay and a target network. Experience replay stores transitions in a circular buffer and samples random minibatches, breaking temporal correlations. The target network is a periodically-copied frozen duplicate of the online network whose parameters are held fixed when computing TD targets, preventing the prediction and target from chasing each other simultaneously.

L(theta) = E [ ( r + gamma * max_{a'} Q(s', a'; theta^-) - Q(s, a; theta) )^2 ]

theta is the online network (differentiated). theta^- is the target network (frozen, treated as a constant). The target network is synchronised to theta every fixed number of steps (e.g., 10,000).

Why it matters: fixing theta^- makes the regression target approximately stationary, converting an unstable coupled system into something resembling supervised learning.

Vanilla DQN uses the same parameters for both selecting the best next action and evaluating it, which causes systematic upward bias because any noise that inflates an action's value also increases the probability of selecting it.

Double DQN decouples the two roles:

target = r + gamma * Q(s', argmax_{a'} Q(s', a'; theta); theta^-)

The online network (theta) selects the action; the target network (theta^-) evaluates it. Errors in the two networks are independent, so the bias largely cancels.

Dueling networks decompose Q into a state value V(s) and a state-dependent advantage A(s, a):

Q(s, a) = V(s) + (A(s, a) - mean_{a'} A(s, a'))

The mean is subtracted to ensure identifiability (V and A cannot otherwise be separated from Q alone).

This is useful because in many states the choice of action barely matters. A shared V(s) head can learn the overall value of a state efficiently without needing to estimate a distinct advantage for every action, leading to better generalisation and more accurate value estimates.

The deadly triad (Sutton & Barto) describes conditions under which Q-learning with function approximation can diverge: combining function approximation + bootstrapping (using network estimates as targets) + off-policy learning. DQN uses all three: a neural network approximator, TD targets derived from the network itself, and a replay buffer that samples transitions from a distribution different from the current policy. In most benchmark environments training is stable, but divergence has been observed and has no fully general theoretical fix.

1. Discrete action space only. DQN selects actions via argmax_a Q(s, a), which requires enumerating all actions. Continuous spaces make this intractable. Remedy: use an actor-critic method (DDPG, SAC, TD3) that maintains an explicit policy network.

2. Exponential blowup when discretising. Naively binning each continuous dimension into k values produces k^d actions for d dimensions, growing exponentially. Even moderate resolution is impractical. The same actor-critic remedy applies.

Prioritised experience replay samples transitions with probability proportional to their absolute TD error (plus a small epsilon for exploration). Transitions the network finds surprising are replayed more often, improving sample efficiency; on 49 Atari games it outperforms uniform DQN on 41 of them.

The correction: because sampling is no longer uniform, the gradient update is biased toward high-error transitions. Importance-sampling weights w_i = (1 / N * 1 / P(i))^beta (annealed from beta < 1 to 1 over training) are applied to each loss term to correct the distribution shift and restore unbiased gradient estimates at convergence.

Regret L(T) is the cumulative difference between the reward collected by an optimal policy and the reward actually collected over T steps: L(T) = sum_{t=1}^{T} [ Q(a) - Q*(a_t) ]. It captures the cost of not knowing which arm is best. A good exploration strategy minimises regret; an algorithm with O(ln T) regret is provably near-optimal for stationary bandits, while fixed-epsilon-greedy accumulates O(T) regret because it never stops exploring bad arms at a constant rate.

UCB1 selects the action maximising Q_t(a) + c * sqrt(ln(t) / N_t(a)), where N_t(a) is the pull count and c is a confidence constant. The second term is a bonus that shrinks as an arm is explored more and grows as time passes without exploring it. This implements optimism in the face of uncertainty: the agent assumes underexplored arms might be excellent until evidence proves otherwise, so it systematically tries them before committing to the greedy choice. The result is O(ln T) regret, provably optimal up to constants for the stationary bandit.

Thompson Sampling maintains a posterior distribution over the true value Q*(a) for each arm. At each step it draws one sample from each posterior and picks the arm with the highest sampled value. In the Bernoulli bandit, conjugate Beta priors make updates exact: after observing reward r from arm a, update Beta(alpha, beta) to Beta(alpha + r, beta + 1 - r). The prior matters most in early steps when data is scarce; a poorly chosen prior (e.g., overly optimistic) can front-load exploration at the wrong arms. Over time the posterior concentrates around the true value and the agent naturally exploits the best arm.

Epsilon-greedy chooses uniformly at random during exploration. In a sparse-reward environment the positive reward may only be reachable via a long, specific sequence of actions; the probability of hitting that sequence by uniform random action is exponentially small. Count-based and curiosity-driven methods address this by adding an intrinsic reward bonus for visiting novel or surprising states, actively steering the agent toward unexplored regions rather than waiting for a lucky random walk to discover distant rewards.

In tabular RL, the visit count N(s) is well defined and the intrinsic bonus 1/sqrt(N(s)) is easy to compute. In deep RL with high-dimensional observations (e.g., Atari pixels), almost every frame is unique, so N(s) = 1 for essentially every state and the bonus is uninformative. Bellemare et al. (2016) replace the count with a pseudo-count derived from a density model: after observing state s, the pseudo-count N-hat(s) is estimated from how much the density model's probability of s changes. This generalises visit frequency to a continuous notion of familiarity without requiring exact state matches.

Curiosity-driven methods reward prediction error of a forward model. A stochastic, uncontrollable stimulus - a TV showing random noise, a fire burning chaotically - produces permanently high prediction error because the noise cannot be reduced through learning. The agent maximises intrinsic reward by staring at the noisy TV indefinitely rather than exploring the task-relevant parts of the environment. The standard mitigation is to compute prediction error only in a learned feature space that encodes agent-controllable factors and discards irrelevant stochasticity, as in Pathak et al. (2017)'s inverse-dynamics encoding.

PPO adds a policy-entropy term c2 * H(pi(. | s_t)) to the training objective, penalising overly peaked action distributions. This keeps the policy from collapsing to a single deterministic action too early in training - essentially keeping epsilon above zero throughout. Unlike UCB, it does not explicitly track uncertainty over action values or state visit counts; it simply prevents premature commitment. The advantage is scalability: the entropy of a softmax head is trivial to compute even for large networks. The disadvantage is that it provides no directed signal toward genuinely unexplored states; it just maintains diversity in the actions taken from already-visited states.

The log-derivative (score function) trick: multiply and divide by \(\pi_\theta\) to convert \(\nabla_\theta \mathbb{E}[R]\) into \(\mathbb{E}[R \cdot \nabla_\theta \log \pi_\theta]\). The gradient now sits inside an expectation over the current policy, so it can be estimated with Monte Carlo rollouts without differentiating through the environment.

Why it matters: this makes policy gradients applicable to any black-box reward signal - no model, no value table, no Bellman backup required.

Actions at time \(t\) cannot affect rewards received before \(t\), so including past rewards in the weight only adds noise. Reward-to-go \(G_t = \sum_{k \geq t} \gamma^{k-t} r_k\) retains the same expected gradient (unbiased) while removing a source of variance. This is sometimes called the causality trick.

Subtracting a state-dependent baseline gives the update weight \((G_t - b(s_t))\). Because \(\mathbb{E}[\nabla_\theta \log \pi_\theta(a \mid s)] = 0\) for any \(b(s)\) that does not depend on \(a\), the baseline contributes zero in expectation: no bias introduced. Its sole effect is to reduce variance by centring the signal, making learning faster and more stable. The optimal baseline is \(V^\pi(s_t)\); using a neural network approximation leads to the actor-critic family.

dist.log_prob(action) computes \(\log \pi_\theta(a_t \mid s_t)\); multiplying by \(G_t\) gives the policy gradient estimator term. Optimisers in PyTorch perform gradient descent (minimisation), but we want gradient ascent on expected return \(J(\theta)\). Negating converts the ascent objective into a minimisation loss. The backward() call then computes \(-\nabla_\theta \log \pi_\theta \cdot G_t\), and the optimiser subtracts this, effectively adding \(\nabla_\theta J\).

1. Sample efficiency: REINFORCE is on-policy - each trajectory is used once. PPO uses importance sampling ratio \(r_t = \pi_\theta / \pi_{\theta_\text{old}}\) to reuse data for multiple gradient steps.
2. Update instability: large gradient steps can collapse the policy. PPO clips \(r_t\) to \([1-\varepsilon, 1+\varepsilon]\), preventing any single update from moving the policy too far even if the gradient points further away. The clipped surrogate objective is cheap to compute and does not require solving a constrained optimisation as TRPO does.

When reward appears only at episode termination, trajectories that never reach a rewarding outcome return a gradient of zero - the policy receives no learning signal at all. Random exploration rarely discovers the rewarding region early in training.

Common remedies:
- Reward shaping: add a dense auxiliary signal (e.g., distance to goal) that guides exploration without changing the optimal policy (if done carefully with potential-based shaping).
- Curiosity-driven exploration: add an intrinsic bonus proportional to prediction error or state novelty, so the agent is rewarded for visiting unfamiliar states even before extrinsic reward arrives.

GAE (Schulman et al. 2016, arXiv:1506.02438) defines a weighted average of \(k\)-step advantage estimates:

where \(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\) is the TD residual.

• \(\lambda = 0\): one-step TD advantage; low variance, but biased by the value function approximation error.
• \(\lambda = 1\): Monte Carlo return minus baseline; unbiased, but high variance.

\(\lambda \in (0,1)\) interpolates smoothly, letting practitioners tune the trade-off. Most modern PPO implementations default to \(\lambda \approx 0.95\), which gives low variance without accumulating too much bootstrap bias.

Because E_{a~π}[∇_θ log π_θ(a|s) · b(s)] = 0 for any fixed b(s). Since b does not depend on the action, it factors outside the expectation; the remaining integral of ∇_θ log π_θ over a normalised distribution is zero (EGLP lemma). The baseline shifts individual sample returns but leaves the gradient's expectation unchanged.

The optimal baseline is b*(s) = V^π(s), the state value function under the current policy. Subtracting it produces the advantage A^π(s,a) = Q^π(s,a) - V^π(s), which is centred around zero and thus minimises the variance of the gradient estimate across actions. In practice V^π is approximated by a learned value network (the "critic" in actor-critic).

Â_t^GAE(γ,λ) = Σ_{l≥0} (γλ)^l δ_{t+l} where δₜ = rₜ + γV(sₜ₊₁) - V(sₜ) is the one-step TD error.

• λ = 0: single-step TD advantage (low variance, higher bias from value approximation error).
• λ = 1: full Monte Carlo advantage (unbiased given perfect V, but high variance).
• λ ∈ [0.9, 0.99] is the typical practical range used in PPO and TRPO, trading a little bias for large variance reduction.

Value-loss gradients flow back through the shared trunk and update feature representations to minimise prediction error for V(s). Those same gradient steps can corrupt or override the features that the policy head has learned to use for good action selection. The two objectives are not aligned: accurate value estimation does not require the same representations as high-entropy, well-calibrated action selection. Common mitigations include separate learning rates, gradient clipping, and a small value-loss coefficient (e.g., 0.5).

Normalisation divides by the batch's standard deviation, which rescales all advantage values to unit variance. If the reward is sparse - most trajectories return zero and a few return a large positive value - the standard deviation is dominated by those rare successes. Normalising inflates the near-zero advantages of the common trajectories and deflates the large positive signal from the rare successes, weakening the very signal the policy needs to learn from. The fix is to use a running statistics normaliser or to clip rather than normalise.

Gₜ (full return from t=0) includes past rewards that the action at step t could not have influenced. Those past rewards add noise with zero mean but nonzero variance to the gradient weight. Replacing Gₜ with the reward-to-go Rₜ = Σ_{t'≥t} γ^{t'-t} r_{t'} discards those causally irrelevant terms, cutting the variance of the estimator without changing its expectation. This is often called the "causality trick" and is the simplest variance-reduction step before introducing a baseline.

If values is not detached, the policy loss gradient flows through the value network as well as the policy network. This means the value network parameters are updated simultaneously by two conflicting signals: one optimising value prediction (value loss) and one optimising policy performance (policy loss). Detaching treats the baseline as a fixed target during the policy update step, cleanly separating the two optimisation problems and preventing gradient interference.

The actor outputs a policy: a probability distribution (or parameters of a distribution) over actions given a state, π_θ(a | s). The critic outputs a scalar value estimate, typically V_φ(s) or Q_φ(s, a). The actor uses the critic's estimate to compute advantages; the critic is trained independently via TD regression.

The advantage subtracts the state-value baseline V(s), which captures the predictable return from being in that state regardless of action. This reduces variance in the gradient estimate without introducing bias (since V(s) does not depend on the action). The resulting signal is a zero-mean, signed quantity: positive means the action was better than the average action from that state, negative means worse.

Lambda controls the exponential decay weighting across TD errors, trading off bias against variance. At λ = 0, GAE reduces to the one-step TD error (low variance, higher bias from the bootstrapped critic). At λ = 1, it reduces to the full Monte Carlo return minus the baseline (unbiased, high variance). Practical values sit around λ ∈ [0.9, 0.97].

If gradients were allowed to flow through the critic's parameters during the actor update, the optimiser would simultaneously try to minimise actor loss by adjusting critic weights. This conflates two distinct objectives and destabilises training. The critic should be a fixed estimator for the purpose of computing the advantage: its current output is used as a target signal, not as a co-optimised component. Detaching breaks the gradient path at that point.

A3C runs multiple independent agent-environment instances in parallel, each asynchronously computing gradients and pushing updates to a shared parameter server. The workers explore different trajectories simultaneously, which decorrelates the gradient updates. This decorrelation provides the same stabilising effect as experience replay in DQN, but without a replay buffer, making the method on-policy and memory-efficient.

The actor and critic impose conflicting gradient pressures on shared layers: the critic loss encourages representations that predict scalar returns, while the policy loss encourages action-discriminative features. If the critic loss coefficient c_v is too large, critic gradients dominate and destroy the policy representation. If too small, the critic trains slowly, leading to poor advantage estimates. The standard fix is careful tuning of c_v (commonly 0.5) or, in high-stakes settings, separate network heads with no shared backbone.

PPO replaces the standard policy gradient objective with a clipped surrogate: L_CLIP = E_t [ min( r_t · A_t, clip(r_t, 1-ε, 1+ε) · A_t ) ], where r_t = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) is the probability ratio between new and old policy. Clipping prevents the ratio from straying too far from 1, bounding the effective policy change per update. This achieves a trust-region-like constraint without the second-order computation required by TRPO.

On-policy: the behaviour policy (the one collecting data) and the target policy (the one being improved) are the same. Off-policy: they differ. SARSA is on-policy; Q-learning is off-policy. Why it matters: off-policy lets you reuse old experience; on-policy cannot.

Q-learning bootstraps off max_a' Q(s', a') (the greedy target policy), so it optimises for the best possible action regardless of what epsilon-greedy exploration actually does. SARSA bootstraps off the action the agent actually takes under epsilon-greedy, including the occasional random cliff-edge step - so its value estimates reflect the exploratory behaviour, penalising risky states. Q-learning's values are cleaner asymptotically but more dangerous during learning.

rho_t = pi(a_t | s_t) / b(a_t | s_t) re-weights a transition collected under behaviour policy b to give an unbiased estimate of the return under target policy pi. Without it, Monte Carlo returns from a different distribution are biased. For multi-step trajectories the ratios multiply, causing variance explosion - which is why algorithms like V-trace clip rho_t to a ceiling c_bar.

1. Sample reuse - transitions stored in a replay buffer can be sampled many times; on-policy data is used once then discarded.
2. Behaviour diversity - you can learn from human demonstrations, scripted explorers, or old checkpoints, as long as the behaviour policy has coverage.
3. Parallelism - actors can asynchronously collect experience while a centralised learner updates the target network (e.g. IMPALA's architecture), decoupling acting speed from learning speed.

The deadly triad (Sutton & Barto) is the combination of: (1) function approximation, (2) bootstrapping (using estimated values to update values), and (3) off-policy updates. Each ingredient alone is fine; together they can cause the value function to diverge rather than converge. DQN tames it with a frozen target network and experience replay, but there is no general convergence guarantee for off-policy deep RL - only empirical stability recipes.

PPO clips the probability ratio r_t(theta) = pi_theta(a|s) / pi_theta_old(a|s) to the interval [1 - epsilon, 1 + epsilon] in its surrogate objective. This prevents the updated policy from drifting too far from the one that generated the rollout batch, so the on-policy assumption is approximately maintained across epochs. Without clipping, repeated epochs on stale data would cause large, destabilising policy updates.

Coverage requires that for every state-action pair the target policy might visit, the behaviour policy must also visit it with non-zero probability: pi(a|s) > 0 => b(a|s) > 0. If violated, the importance sampling ratio becomes undefined (division by zero), and the replay buffer contains no gradient signal for those transitions. The agent silently extrapolates from neighbouring states, often catastrophically - a particular hazard in robotics manipulation where rare, precise contacts matter most.

It adds a bonus proportional to the Shannon entropy of the policy at each visited state: r_t + α · H(π(·|s_t)). The agent is rewarded not just for collecting reward but for acting unpredictably. This prevents premature collapse to a single deterministic action and keeps exploration alive throughout training.

The hard Bellman backup uses max_a Q(s,a) for the target value. The soft Bellman backup replaces this with the soft maximum:

V_soft(s) = α · log Σ_a exp( Q(s,a) / α )

The soft maximum is smooth and differentiable, always less than or equal to the hard maximum, and approaches the hard maximum as α → 0. This makes gradient-based optimisation more stable and encodes the entropy bonus directly into value estimates.

The optimal policy is a Boltzmann (softmax) distribution over actions:

π*(a|s) ∝ exp( Q*(s,a) / α )

High-Q actions receive higher probability, but probability mass is never zero on any action. Temperature α controls the spread: large α approaches uniform; small α approaches greedy. This is exactly the softmax policy familiar from multi-armed bandits, extended recursively through the Bellman equation.

SAC frames temperature selection as a constrained optimisation: maximise expected reward subject to the constraint that the policy's mean entropy stays above a minimum target H_target (often set to -|A|, the negative of the action-space dimension). The Lagrangian dual variable for this constraint is α. Gradient descent on the dual variable automatically raises α when entropy falls below the target and lowers it when entropy exceeds the target, making temperature self-tuning during training.

PPO adds an entropy bonus as a coefficient c₂ · H(π_θ) directly to its surrogate loss, acting as a soft regulariser to slow policy collapse. It does not modify the Bellman backups. SAC bakes entropy into the value function through the soft Bellman backup, so the exploration incentive persists throughout every value estimate. PPO's entropy term primarily helps in the early phases of training on discrete action spaces; SAC's entropy shaping is structurally deeper and more persistent.

1. Reward-sparse tasks: a high-entropy policy may never encounter reward, so the entropy bonus dominates and the agent learns to act randomly rather than solve the task. Pre-training with curriculum or reward shaping is needed.

2. Unbounded Gaussian variance: in continuous action spaces, a Gaussian policy can inflate its variance indefinitely in low-reward regions to harvest entropy. Without variance clamping or squashing (e.g., tanh), this causes diverging variances or NaN gradients. SAC addresses this with a log-variance parameterisation and squashing, but naive implementations fail here.

α scales the relative weight of entropy versus reward in the objective. At α → 0 the entropy bonus vanishes and the policy collapses to the standard reward-maximising (greedy) policy. At α → ∞ the reward signal is negligible and the optimal policy is uniform over all actions. Intermediate values of α produce stochastic policies that concentrate more probability on higher-reward actions while retaining spread across alternatives, balancing exploitation and exploration.

The problem of determining which actions in a long sequence were responsible for a delayed reward. When a reward arrives many steps after the actions that caused it, the agent must propagate credit backward through time to assign appropriate value to each decision. First formalised by Minsky in 1961, it remains a core challenge in RL.

The bonus must be potential-based: \(F(s, a, s') = \gamma \Phi(s') - \Phi(s)\) for some potential function \(\Phi: \mathcal{S} \to \mathbb{R}\). This telescoping form sums to zero over any complete trajectory, so the agent's ranking of policies under the shaped reward is identical to its ranking under the original reward.

TD(\(\lambda\)) maintains eligibility traces that keep recent states and actions "eligible" for credit updates. Each step back, eligibility decays by \((\gamma\lambda)^k\). At \(\lambda = 0\) you get one-step TD (minimal propagation); at \(\lambda = 1\) you recover full Monte-Carlo returns (credit reaches all the way back). Larger \(\lambda\) assigns credit further back but increases variance; smaller \(\lambda\) is lower variance but propagates credit more slowly.

GAE constructs the advantage estimate as \(\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^l \delta_{t+l}\), a \(\lambda\)-weighted blend of multi-step TD errors. The baseline \(V(s_t)\) subtracted inside each \(\delta\) removes the expected return regardless of action, so only the action-specific contribution remains. This reduces gradient variance substantially while the \(\lambda\) parameter lets practitioners dial in the bias-variance trade-off.

The policy gradient theorem states \(\nabla_\theta J = \mathbb{E}[\nabla_\theta \log \pi_\theta(a|s) \cdot G_t]\). Any baseline \(b(s)\) that depends only on state can be subtracted without changing the expectation, because \(\mathbb{E}[\nabla_\theta \log \pi_\theta(a|s) \cdot b(s)] = 0\). Subtracting \(V(s_t)\) gives the advantage \(A(s,a) = Q(s,a) - V(s)\), which directly answers "how much better was this specific action than average?" - a credit assignment signal centred on zero.

RUDDER (Arjona-Medina et al. 2018) trains a sequence model (e.g. an LSTM) to predict the cumulative return from a trajectory prefix. The contribution of each new observation to the prediction - the change in predicted return at each step - is used as a redistributed per-step reward. The method requires a sequence model accurate enough that its prediction updates correspond to genuine action contributions, not noise. Under perfect prediction, the redistributed rewards reduce expected future return to zero, eliminating discounting bias and simplifying Q-value estimation.

1. Potential mispecification: A potential function that points away from the true goal along some paths makes learning slower, not faster. Potential-based shaping preserves optimality but does not guarantee faster convergence when the heuristic is misleading.

2. Stochastic long-horizon environments: When outcomes are partly due to chance, eligibility traces and return redistribution conflate skill with luck. Credit assigned to actions that happened to precede a lucky outcome introduces noise into the value estimates, destabilising training.

On reflexive single-step tasks - "what is the capital of France", "is this sentence positive or negative", short classification. Forced reasoning adds latency, increases token cost, and can lead the model astray by inventing spurious justifications. Reserve CoT for multi-step problems where intermediate state genuinely matters (maths, logic, planning).

Sample multiple reasoning chains for the same prompt (5-40), then take the majority answer. Different chains stumble in different places but converge on the correct answer more often than any single chain. Accuracy lifts on hard reasoning benchmarks are typically 5-20 points - far larger than most prompt tweaks deliver, at the cost of N-times inference.

Not quite. Frontier models (Claude, GPT-4 class, Gemini) trained on reasoning traces produce CoT implicitly, and some expose explicit "thinking" tokens (Claude extended thinking, OpenAI o-series). Explicit CoT instructions still help on edge tasks, weaker open-source models, and when you want to inspect the reasoning. The shift is that you no longer need "Let's think step by step" to unlock the capability - you choose whether to surface it.

1. Autoregressive (next-token): hides the next token, reads left context only (causal). GPT family.
2. Masked LM (MLM): hides ~15% of tokens at random, reads both directions. BERT.
3. Span corruption: hides contiguous spans replaced by sentinels, encoder reads both sides and a decoder regenerates them. T5.

What an objective hides decides what signal the model extracts and what it can later do.

The training task and the inference task are identical: you train by predicting the next token and you generate by predicting the next token, so there is no train-test gap. MLM trains on a 15%-masking corruption pattern that never occurs at generation time, so making a fluent generator out of it means fighting its own objective. Autoregression also forces broad competence (syntax, facts, arithmetic) because all of it reduces next-token uncertainty.

No. A freshly pretrained base model is a next-token predictor; asked a question it may just continue with more questions. SFT plus preference optimisation (RLHF/DPO) turn it into an assistant, but they only surface and direct capability the pretraining objective already instilled. They redistribute probability mass; they do not teach a skill the pretraining loss never rewarded.

The model trains on ground-truth prefixes but generates on its own (possibly wrong) prefixes, so an early mistake shifts the model off the distribution it was trained on and errors can compound across a long generation. It is one reason decoding strategy matters and why long generations drift. The objective optimises one-step-ahead likelihood, not multi-step rollout quality.

When the inference task is not open-ended generation. Bidirectional MLM encoders produce the strongest embeddings for retrieval and classification; encoder-decoder span-corruption models (T5) stay competitive where there is a clear input-to-output mapping (translation, summarisation). Match the objective to what the model must do at inference, not to whichever sounds more powerful.

The loss does not care what it compresses. The same next-token objective on web sludge versus curated tokens produces very different models, but nothing in the objective flags the difference; the corpus does the quiet work. This is why deduplication, filtering, and curation are first-class levers, not afterthoughts.

LLMs have a fixed training cutoff and no access to private documents. RAG solves this by retrieving relevant information at query time and injecting it into the prompt, making the model a reasoning layer over externally supplied data.

1. Index - chunk documents and store their embeddings in a vector database. 2. Retrieve - embed the user query and find the top-k most similar chunks. 3. Generate - place retrieved chunks in the prompt and instruct the model to answer from that context only.

Chunk size directly affects retrieval quality. Chunks that are too small lose surrounding context, while chunks that are too large introduce irrelevant text that dilutes the signal. There is no universal optimal size - it must be determined by evaluation on your specific data.

Embedding model quality matters more. A high-quality embedding model running on a simple store like SQLite will outperform a mediocre embedding model on a purpose-built vector database like Pinecone.

Models exhibit recency bias, placing disproportionate weight on content near the end of long contexts. To counteract this, place the most important retrieved chunks last in the prompt.

Production RAG commonly adds query rewriting (to handle conversational follow-ups), hybrid search combining vector and BM25 retrieval, cross-encoder re-ranking (reducing top-50 chunks to top-5), and per-source citation. Each addition should be justified by measurable improvement in evaluations.

1. Ingestion - split docs into 200-800 token chunks with 50-100 token overlap.
2. Embedding - encode each chunk and store vectors in an index.
3. Retrieval - embed the query, return top-k nearest chunks.
4. Reranking - rescore top-k with a cross-encoder (optional but high-leverage).
5. Generation - inject chunks into the prompt and ask the model to answer.

Skipping reranking is the single biggest quality win most teams leave on the table.

LLMs attend most strongly to the start and end of long contexts and skim the middle. If your top-1 retrieved chunk sits in the middle of ten chunks, the model often ignores it. Reorder retrieved chunks so the most relevant sit at the prompt edges (highest at the end is a strong default for decoder-only models). Cheap to implement, measurable accuracy lift.

Pure vector search misses exact-keyword matches (product codes, error strings, proper nouns). Pure BM25 misses semantic paraphrases. Combine the two ranked lists with Reciprocal Rank Fusion (RRF) and you get the best of both with almost no extra latency. This is now a default rather than an optimisation.

• Fixed-token chunks that slice through sentences mid-clause.
• No overlap, so context that straddles boundaries vanishes.
• Chunks too small (lose context) or too large (dilute the signal in cosine).

Default to recursive splitting on paragraph -> sentence -> token boundaries, 400 tokens with 50-100 overlap. Tune from there.

This is hallucinated citation: the model produces text in the shape of a quote but invents the content. Mitigations:

1. Instruct it to quote verbatim from chunks and mark the chunk ID.
2. Post-hoc verify quoted spans actually appear in the retrieved chunks.
3. For high-stakes outputs, reject answers whose quotes fail verification.

Never trust a citation just because it is formatted like one.

As d_k grows, the variance of Q . K^T grows linearly, pushing some logits into regions where softmax saturates and gradients vanish. Dividing by sqrt(d_k) rescales the dot products to unit variance, keeping the softmax in its responsive range. Without it, training a deep transformer becomes numerically fragile and convergence stalls.

Self-attention: Q, K, V all derived from the same sequence (used inside encoder blocks and inside decoder blocks). Cross-attention: Q from one sequence, K and V from another (the bridge from encoder to decoder in encoder-decoder models). The math is identical; only the input plumbing differs.

It is O(n^2) in both compute and memory. Doubling context quadruples the attention matrix, so by 100k tokens attention dominates the forward pass and the KV cache blows out GPU memory. FlashAttention reorders the computation to stay memory-efficient; Longformer/BigBird sparsify; Mamba and RWKV drop attention for recurrent state. Pick the variant that matches your context-length and recall requirements.

Each head learns a specialised type of dependency - one tracks syntactic agreement, another resolves coreference, another acts as a positional nearest-neighbour. Running them in parallel lets the layer compose several relations at once, and stacking layers composes them into hierarchical structure. The insight: depth gives you composition, heads give you parallel specialisation.

1. Multi-head self-attention.
2. Residual connection + layer norm wrapping the attention.
3. Feed-forward MLP (typically 4x model dimension wide).
4. Residual connection + layer norm wrapping the MLP.

GPT-2 stacks 12, GPT-3 stacks 96, GPT-4-class models stack 80-120. The block is the unit of scale.

• Encoder-only (BERT): bidirectional attention, no generation. Best for classification, NER, embeddings.
• Decoder-only (GPT family): causal mask, autoregressive generation. The default for chat, completion, and most LLMs today.
• Encoder-decoder (T5): encoder reads input fully, decoder generates conditioned on it. Still strong for translation and summarisation where input and output are clearly separated.

If in doubt for an LLM-style product, pick decoder-only.

Rotary position embeddings encode position by rotating query and key vectors in 2D subspaces, so relative position falls out of the dot product naturally. This generalises far better to sequences longer than training context than learned absolute embeddings, and it composes cleanly with FlashAttention. Llama, Mistral, Qwen and PaLM all use it; learned absolute embeddings are mostly legacy at this point.

Position is injected explicitly, either by adding sinusoidal vectors to the embeddings (original paper), learning a positional embedding table (BERT, GPT-2), rotating Q/K vectors (RoPE - Llama, Mistral), or adding a linear bias to attention logits (ALiBi - Bloom). Strip positional info entirely and the model collapses to a bag-of-words.

Starting from a byte alphabet, BPE iteratively merges the most frequent adjacent pair of symbols until the vocabulary hits a target size (typically 30k-200k), so common fragments become single tokens and rare strings decompose into smaller pieces.

BPE merges are learned from the training corpus, which is dominated by English. Under-represented scripts and languages never get long merges, so each word fragments into many small tokens (sometimes one per byte of UTF-8). That inflates input cost, output cost, and effectively shrinks the context window. It is the quietest tax in the API bill.

Dump the tokenisation of the numbers. Older GPT tokenisers split numbers as multi-digit chunks in unpredictable ways ("1234" might become "12" + "34", "123" + "4", or four single digits depending on context). The model is not bad at arithmetic - it is solving a different arithmetic problem on each input. Many "weird model behaviour" bugs hide in unexpected token splits.

Both implement BPE-style subword tokenisation. SentencePiece (Llama, T5, PaLM) treats whitespace as a regular character so it works language-agnostically without pre-tokenisation. Tiktoken (GPT, Claude) is highly optimised in Rust and assumes whitespace-separated languages. The vocabulary differs; the BPE skeleton is the same.

Cosine ignores vector magnitude and compares direction only, so it is robust to differences in sentence length or model calibration. Range is a clean -1 to 1. When embeddings are L2-normalised (almost always), cosine equals the dot product, which is what HNSW and pgvector actually compute under the hood. Euclidean distance correlates poorly with semantic similarity in high dimensions and is rarely used.

Around 100k vectors. Below that, brute force is a one-liner and fast enough. Above that, latency creeps over 100ms and you reach for HNSW (pgvector, Qdrant), IVF (FAISS), or LSH. The cost is a small recall hit - tune index parameters to trade recall against speed.

1. Domain-adapt the model. Fine-tune a sentence-transformer on your own query/passage pairs, or pick a model already trained for the domain (legal-BERT, BioBERT, FinBERT).
2. Use asymmetric retrieval. Different encoder for queries vs documents (E5-asym, BGE-large with query/passage prompts), because short queries and long docs occupy different regions of vector space.

The pitfall is assuming all-MiniLM-L6-v2 works everywhere; it was trained on generic web text.

Take a transformer encoder, pool its hidden states (mean-pool or [CLS]) into a single vector, then train with a contrastive objective: pull semantically similar pairs together and push dissimilar pairs apart. Sentence-Transformers, E5, BGE and Voyage all follow this recipe. The contrastive loss is what turns "language model" into "similarity geometry".

Raw self-attention is permutation-invariant: it treats the input as a set, so permuting the tokens just permutes the output and "the dog bit the man" is indistinguishable from "the man bit the dog". Order is not in the attention math; it has to be added, either by encoding position into the embeddings (sinusoidal, learned, RoPE) or by biasing the scores (ALiBi). Everything a model knows about word order, it knows because position was injected somewhere.

RoPE rotates query and key vectors by an angle proportional to position. In the dot product the absolute angles cancel and only the relative distance (m - n)*theta survives, so the attention score depends on how far apart two tokens are, not where they sit absolutely. Relative distance is the quantity that transfers past the training length. A learned absolute table, by contrast, simply has no row for a position it never trained on.

ALiBi adds no embeddings. It subtracts a penalty from each pre-softmax attention score, linear in query-key distance and scaled by a fixed per-head slope, so each head attends to a soft recency-biased window. Because the bias is purely a function of distance with no length-bound parameters, it extrapolates past training length ("train short, test long"). The trade-off: the monotonic recency penalty makes it hard to attend sharply to a single distant token, which hurts retrieval-style long-context tasks.

For length generalisation in small decoder-only models, no positional encoding at all (NoPE) matched or beat ALiBi and RoPE. It works because the causal mask already breaks permutation symmetry: token i can attend only to j <= i, so the model can recover position by counting visible tokens, and it learns relative-style attention without being told. The takeaway is not to delete positional encodings (frontier models keep RoPE for sharper control at scale) but that position is partly emergent from causality.

Push a RoPE model past its training length and the high-frequency dimension pairs rotate into angles never seen, so local syntax corrupts first. Position interpolation rescales long positions back into the trained range; YaRN refines this by interpolating slow long-range frequencies while leaving fast local ones nearly untouched, recovering most quality with roughly an order of magnitude less fine-tuning. The relative-distance property is exactly what makes this remapping coherent.

At each step it emits a probability distribution over the whole vocabulary. A separate decoder turns that distribution into an actual token. Swap the decoder and the same frozen weights swing from dull looping text to vivid prose to confident nonsense. Many "the model is bad at X" complaints are really decoder-settings complaints, fixable by tuning two numbers rather than re-running a fine-tune.

Beam search maximises total sequence probability. On open-ended tasks the highest-probability sequence is bland, repetitive, and degenerate, because high likelihood and high quality diverge for creative text. Beam shines where there is one mostly-correct answer (translation), but for story or chat generation it reads like a hostage note. There you want sampling, not maximisation.

Temperature T divides the logits before softmax: T < 1 sharpens toward top tokens, T > 1 flattens, T -> 0 is greedy. Used alone it is blunt because raising T for diversity also inflates the long tail of genuinely bad tokens, so you get creativity and incoherence together. That is why temperature is almost always paired with a truncation step (top-p or min-p) that cuts the tail first.

• top-k: the k highest-probability tokens. Fixed size, so it is mis-sized when the distribution is very peaked or very flat.
• top-p (nucleus): the smallest set whose cumulative probability exceeds p. The shortlist adapts to model confidence.
• min-p: every token with probability at least min_p * p_max. The floor scales with the top token, staying coherent even at high temperature.

Sampling noise compounds across a long generation, and one wrong token early can derail an entire chain of thought or break code. Reasoning-tuned models are typically decoded near-greedy (low temperature) because here diversity buys nothing and costs correctness. The deliberate exception is self-consistency, which re-introduces sampling to draw several diverse chains and majority-vote the answer.

Query represents what a token is looking for, Key represents what a token matches against, and Value is what a token contributes when matched. Together they enable each token to selectively gather information from other tokens based on relevance.

The formula is: attention(i) = softmax(Q_i · K^T / sqrt(d)) · V. Dividing by sqrt(d) prevents the dot products from growing too large as dimensionality increases, keeping gradients well-scaled during training.

RNNs process tokens sequentially, making long-range dependencies hard to maintain over many steps. Attention allows every token to directly attend to every other token in one operation, regardless of distance in the sequence.

Attention is O(n²) in sequence length because every token must attend to every other token. A 100k-token context requires roughly 10 billion attention scores, making long-context computation a significant engineering challenge.

Multi-head attention runs several attention operations in parallel, each potentially learning a different type of relationship such as syntax, coreference, or positional adjacency. A single head can only capture one relational pattern at a time, whereas multiple heads allow richer, diverse representations.

Softmax converts the raw dot-product scores between a query and all keys into a probability distribution that sums to 1. This ensures the weighted combination of Value vectors is a proper weighted average, highlighting the most relevant tokens.

Each token produces a Query (what am I looking for?), a Key (what do I match against?), and a Value (what do I contribute if matched?). These three vectors enable tokens to compare themselves against all other tokens and extract weighted information.

The formula is: attention(i) = softmax(Q_i * K^T / sqrt(d)) * V. The dot products measure relevance between the query and all keys, softmax converts those scores into a probability distribution, and the result is a weighted sum of values.

Dividing by sqrt(d) - where d is the dimension of the key/query vectors - prevents the dot products from growing too large as dimensionality increases. Without this scaling, softmax outputs become extremely peaked and gradients vanish during training.

Attention lets every token directly attend to every other token in a single operation, regardless of distance. RNNs process tokens sequentially, so information from distant tokens must survive many steps of compression, causing it to degrade.

Self-attention is O(n^2) in sequence length because every token attends to every other token. For a 100k-token context this means roughly 10 billion attention scores, making long-context inference extremely expensive in memory and compute.

Human annotation of large fine-tuning datasets is prohibitively expensive (tens of cents to tens of dollars per example for expert tasks). A capable generative model can produce the same volume of labelled examples orders of magnitude more cheaply, collapsing the annotation bottleneck. The value proposition holds as long as the generator's errors are manageable.

Instruction synthesis generates an instruction-response pair and accepts it as-is. Rejection sampling generates multiple candidate responses for each prompt and discards those that fail a verifier (unit test, ground-truth answer check, classifier). The result is a training set composed only of correct or high-quality outputs, even when the base model's per-sample pass rate is low. This is especially useful for reasoning tasks where correctness can be verified mechanically.

Stage 1 (SL-CAI): the model generates a response, then critiques and revises it according to a written list of principles (the "constitution"). The revised response becomes a supervised fine-tuning target. Stage 2 (RL-CAI / RLAIF): the model generates preference pairs (original vs. revised), which train a preference model. Policy training then uses this AI-generated preference signal rather than human annotations. Humans only author the constitution once; all subsequent labelling is done by the model itself.

Model collapse (also called Model Autophagy Disorder) is the progressive loss of output quality or diversity when a model is trained iteratively on its own generated data without sufficient injection of real data. Each generation's errors and distributional biases are amplified in the next. The result is a model concentrated on high-probability outputs, losing the ability to produce rare but correct responses. The key cause is the absence of a grounding signal from real data to counteract drift.

phi-1 was trained on GPT-3.5-generated "textbook quality" code exercises (roughly 1B synthetic tokens) rather than raw web-scraped code. Textbook-style data is informationally denser: it explains concepts with build-up, worked examples, and commentary, mirroring how code is taught rather than how it appears in production. This suggests that data quality and pedagogical structure can substitute for scale when the task is well-defined.

Goodhart's Law states that when a metric is used as a training target, it ceases to be a good measure of the underlying goal. In RLAIF, if the policy model and the critique model share architecture or training lineage, the policy can learn to produce outputs that score well on the critique without genuinely improving in harmlessness or helpfulness. The critique score diverges from actual quality. This is why constitutional AI requires careful design of the critique model and regular re-evaluation against human preference ground truth.

1. No correct samples exist: if the base model never generates a correct answer for a hard problem class (e.g., a proof the model lacks the knowledge for), rejection sampling produces no accepted examples for that class - the filter keeps nothing. 2. No reliable verifier: if correctness cannot be determined mechanically (e.g., open-ended creative writing, nuanced ethical reasoning), there is no filter to apply, and all candidates are accepted indiscriminately, defeating the purpose.

1. Seed pool - start with a small set of human-written tasks (175 in the original paper).
2. Instruction generation - the model samples a few tasks from the pool and writes a new instruction.
3. Instance generation - the model produces input-output pairs for each new instruction.
4. Filtering - near-duplicates and degenerate samples are removed (ROUGE-L similarity threshold of 0.7); survivors re-enter the pool.

The loop then repeats, expanding the pool without additional human annotation.

The ROUGE-L diversity filter catches redundancy but has no mechanism to verify factual accuracy. The model generates outputs that are plausible in form but may be wrong in substance, and those wrong-but-well-formatted answers pass the filter. The model's existing capability ceiling determines the quality ceiling; it cannot reliably produce correct outputs for tasks already beyond its competence.

In the original Self-Instruct, the same model serves as both generator and student (true self-improvement). In Alpaca, a stronger teacher model (text-davinci-003) generates the 52,000 demonstrations while a weaker student (LLaMA-7B) is fine-tuned on them. That is knowledge distillation, not self-improvement. The capability ceiling is the teacher's, not the student's, but this also introduces licence/IP complications and means errors are subtler and harder to filter.

1. Capability ceiling - the student can only learn up to the teacher's competence level.
2. Licence/legal exposure - generating training data from a proprietary API to train a competing model may violate terms of service (a concern the Alpaca authors flagged explicitly).
3. Subtler error propagation - a stronger teacher produces well-formatted but factually wrong outputs that pass mechanical filters more easily than crude errors would.

A model trained on Self-Instruct data may learn the stylistic surface features of instruction responses (bullet-point formatting, hedging phrases like "Sure, here is...") without learning the underlying task semantics. This produces models that sound confident and well-structured but give incorrect or shallow answers. It is the mechanism behind much of the sycophantic, over-hedged tone observed in early chat-tuned models.

ROUGE-L filtering removes near-duplicate strings, but it does not force the model to venture into low-confidence task types. Because the generator samples from its own distribution, tasks the model handles confidently are generated frequently; tasks requiring rare or specialised knowledge are rarely generated at all. Over many iterations the pool gravitates toward a subset the model is already good at, starving the fine-tuned model of training signal for hard or specialised tasks.

• Self-Instruct (Wang et al., 2022): fine-tuning GPT-3 with the pipeline produced a 33 percentage-point absolute improvement over the base model on Super-NaturalInstructions, approaching InstructGPT-001 performance.
• Alpaca (Stanford, 2023): in blind human comparisons against text-davinci-003, the 7B Alpaca model won 90 comparisons to 89, suggesting rough parity despite being far smaller and cheaper to produce (data generation cost under $500).

Seed instruction datasets (e.g., Alpaca) concentrate at low complexity - simple recall and formatting tasks. Evol-Instruct uses an LLM to iteratively rewrite seed instructions into progressively harder variants, producing a training dataset whose difficulty distribution spans simple to highly constrained multi-step tasks. Models fine-tuned on this spread generalise far better to complex real-world prompts.

1. Add constraints - introduces extra conditions the answer must satisfy.
2. Deepening - shifts the instruction toward a more specialised sub-problem.
3. Concretising - replaces vague terms with specific technical vocabulary.
4. Increased reasoning - demands multi-step inference rather than direct recall.
5. Complicate input - makes the input artefact itself more intricate (longer code, nested data).

These are applied stochastically; each evolved instruction is produced by one randomly sampled operator.

Depth evolution makes an existing instruction harder while keeping the same topic. Breadth evolution generates a brand-new instruction on a related but distinct topic, inspired by but different from the original seed. It expands subject-matter coverage rather than difficulty within a fixed subject. Both types of evolved instructions are included in the final training mix.

The filter (implemented as a second LLM judge call) discards outputs that:
- Repeat the original instruction with only cosmetic changes.
- Contain meta-text from the rewriting prompt (e.g., "I will now make this harder by...").
- Are incoherent or unanswerable due to conflicting constraints.
- Are empty strings (often from refused instructions).

Without filtering, corrupted examples enter training directly. The filter is also where most of the pipeline's inference cost concentrates, since every candidate needs a separate judgment call.

WizardCoder adapted the depth operators for code: evolving instructions to require higher algorithmic complexity, edge-case handling, or resource constraints. Starting from Code Alpaca, it achieved strong HumanEval pass@1 scores (ICLR 2024).

WizardMath added a reinforcement learning stage (RLEIF): evolved instructions are used for SFT and as RL prompts scored by a process reward model. This produced a 7B model competitive with much larger models on GSM8k and MATH (ICLR 2025 Oral).

Both show that Evol-Instruct is domain-agnostic when operators are adapted to domain-specific notions of difficulty.

The evolved instructions carry the stylistic fingerprint, knowledge gaps, and refusal patterns of the LLM used to generate them (GPT-4 in the original WizardLM work). A student model fine-tuned on this data learns to mimic those biases, not just to follow complex instructions. This is a form of distribution shift: the training signal conflates "harder instruction following" with "behaving like the generator model," which can impair diversity and introduce subtle systematic errors.

The only quality signal is the elimination filter, which is itself an LLM judge. LLM judges carry systematic biases (e.g., preferring verbose or superficially complex text) that propagate silently into the training set. There is no external oracle - unlike mathematical reasoning tasks where a symbolic checker can verify correctness. This means quality collapse at high evolution depth is hard to detect, and cascading errors across multiple rounds can produce chains of plausibly-formatted but subtly broken training examples that pass the filter.

Hard labels (one-hot) encode only the correct class and discard every other class relationship. Soft labels carry the teacher's full output distribution, encoding which "wrong" answers the teacher considers plausible and how similar different classes are. This richer gradient signal lets the student learn inter-class structure, not just which label is correct.

Why it matters: the teacher's soft distribution is itself compressed knowledge about the input space; training against it is far more data-efficient than training against human annotations alone.

Temperature T divides logits before the softmax: p_i = exp(z_i / T) / sum(exp(z_j / T)).

• T = 1: standard softmax; probability mass is concentrated near the argmax, making near-zero entries nearly invisible.
• T > 1: distribution "softens"; probability mass spreads over more tokens, amplifying the gradient signal from low-probability but informative entries.

Both teacher and student use the same T during distillation. After training, T is reset to 1 for inference. Common values are T = 2-10 for classification; in LLM distillation T is often kept low (1-2) because the vocabulary is already large.

1. Output-token distillation - student trains on decoded text from the teacher. Cheapest: only final responses needed. Example: Alpaca (Stanford, 2023).

2. Soft-distribution distillation - student minimises KL divergence against the teacher's per-token vocabulary distribution. More expensive: requires teacher logits at every position, stored or streamed.

3. Reasoning-trace distillation - teacher generates chain-of-thought explanations; student learns the problem-decomposition strategy, not just the answer. Most expensive to generate, but transfers the deepest capability. Example: Orca (Microsoft, 2023).

Cost increases from 1 to 3; depth of transfer also increases from 1 to 3.

Imitation models trained purely on teacher output text learn to mimic the teacher's discourse style far more readily than its factual depth or reasoning ability. Human raters consistently rated these models as competitive with ChatGPT; automated benchmarks on out-of-distribution factual and reasoning tasks revealed a persistent capability gap.

The "false promise" is that surface fluency and polished phrasing create the illusion of transferred capability. The student absorbs how the teacher writes, not what the teacher knows. This motivates supplementing output-token distillation with richer signals (logits, reasoning traces) and evaluating on tasks outside the distillation distribution.

Model collapse (or "going MAD" per Alemohammad et al., 2023) occurs when models are trained repeatedly on outputs from previous model generations without injecting fresh real-world data. Each generation fits a compressed version of the previous one's distribution, systematically losing the long-tail variance present in real data. Precision (output quality) may initially hold while recall (diversity) degrades; eventually both collapse.

Root cause: generative models oversample the high-probability modes of their own output distribution. Iterative training amplifies this bias, narrowing the effective support of the learned distribution.

Mitigation: continually mix in real human-generated data alongside synthetic data; never train solely on self-generated outputs across multiple rounds.

Standard instruction fine-tuning teaches the student to produce correct final answers. Orca trained on GPT-4's step-by-step reasoning traces, making the intermediate problem-decomposition strategy observable to the student. The student learned:

1. How to break complex problems into sub-problems.
2. When to apply which reasoning pattern (analogy, deduction, retrieval, etc.).
3. Self-correction patterns the teacher exhibited mid-trace.

A 65B model trained on output tokens alone had been taught "what to say"; a 13B model trained on explanation traces was taught "how to think about it." The additional supervision signal from reasoning steps more than compensated for the 5x parameter gap on structured reasoning benchmarks.

Rejection sampling generates k candidate responses per prompt from the teacher, then filters to keep only those passing a quality criterion (unit tests for code, exact-match on maths answers, a judge model score for open-ended tasks). Only the accepted responses enter the student's training set.

Why apply it: even a highly capable teacher produces errors, especially on long reasoning chains. Training the student on those errors teaches it to confidently reproduce the teacher's failure modes. Filtering by verifiable correctness before training substantially improves the signal-to-noise ratio in the distillation set, yielding a better student than training on unfiltered teacher outputs.

The accepted responses also tend to be the teacher's most coherent outputs, giving the student a higher-quality distribution to fit.

Rejection sampling discards any teacher-generated trace whose final answer does not match the gold label, keeping only traces that led to a correct result. This ensures the student model trains on reasoning paths that are at least outcome-correct, preventing erroneous intermediate steps from being directly supervised.

Why it matters: Without this filter, wrong traces degrade student performance relative to plain answer-only fine-tuning.

A small model trained only on (question, answer) pairs must infer the latent reasoning strategy from the answer signal alone - a very sparse gradient that the model may lack the capacity to resolve into reliable intermediate steps. Providing the full trace as a training target converts this discovery problem into a standard sequence prediction task: the model learns to reproduce the explicit steps, and correct reasoning behaviour emerges as a consequence.

During fine-tuning, the student always receives teacher-quality trace prefixes (clean, correct intermediate steps). At inference the student must condition on its own, lower-quality trace prefixes. If the student's trace degrades midway through a problem, it has never trained to recover from its own errors as context, so final-answer quality drops sharply. This is a specific instance of the more general exposure bias in autoregressive training.

Basic RFT keeps traces that produced correct answers and discards the rest - the trace is only a means to verify correctness. Orca explicitly prompts GPT-4 to generate explanation traces and step-by-step thought processes as first-class training targets, regardless of whether the answer could have been verified by other means. The student learns from three signal layers simultaneously: the explanation, the reasoning process, and the final answer. This richer supervision is why a 13B Orca model can surpass much larger models on complex benchmarks.

When a model generates its own fine-tuning data and retrains on it repeatedly, low-probability but valid reasoning strategies are increasingly unlikely to appear in the sampled traces. Each round narrows the distribution toward the model's dominant, high-confidence paths. Over several iterations the model becomes confidently wrong on problems that require the rarer strategies it has stopped generating. This is the reasoning-specific form of mode collapse in synthetic data loops.

Fine-tuning on a large quantity of domain-specific traces (e.g., maths reasoning) shifts the model's weights toward the representations and output patterns that are rewarded in that domain. Parameters previously used for other tasks are repurposed. Fu et al. (2023) measured this directly: models distilled on maths reasoning traces improved on GSM8K but degraded on language-understanding benchmarks. The trade-off is acceptable for narrow deployment targets but costly for general-purpose assistants.

The student's ceiling is the teacher's reliability on that task. Rejection sampling only keeps traces that led to correct answers, so if the teacher solves a class of problems at 60% accuracy, the fine-tuning set for that class is sparse and potentially unrepresentative. On tasks where even frontier teachers fail frequently, the distillation pipeline produces too little or too low-quality data for meaningful transfer. The student cannot learn a reasoning strategy the teacher does not reliably demonstrate.

Sample k completions per prompt from the current model, filter them through a verifier (ground-truth checker or reward model), and fine-tune on the completions that pass. The model's own outputs replace human-written training labels. Yuan et al. (2023) moved a 7B LLaMA from 35.9% to 49.3% on GSM8K using this approach with k=100.

When the model fails a question, STaR prompts it: "Given that the answer is X, construct a reasoning trace that leads to X." The resulting rationale is added to the training set even though the model could not reach X unprompted. Without rationalisation, only already-solvable problems accumulate training data, permanently excluding hard problems from improving the model.

Full REINFORCE updates on both correct and incorrect samples: ∇L = E[r(y) · ∇ log π(y|x)]. RFT approximates this by using binary reward and discarding failed samples rather than penalising them. It gains simplicity (no critic, no KL penalty machinery) but loses: (1) explicit negative signal from near-misses, and (2) policy drift correction - the distribution can shift substantially across iterations without a reference-model KL guard.

If k=100 samples per prompt all follow the same reasoning path, the model receives 100 copies of nearly identical data - it is like training on one example 100 times. The fine-tuning signal comes from learning different routes to a correct answer. Yuan et al. found that pooling rejection samples across multiple model checkpoints or temperatures outperformed collecting more samples from a single model.

If the model's probability of generating a correct solution is near zero, sampling any finite k yields no passing completions, so there is no data to train on. The loop stalls. Mitigations: (1) start from a stronger base model, (2) use a teacher/larger model to seed the initial correct solutions (distillation first, then self-improvement), or (3) decompose hard problems into simpler sub-tasks the model can already solve.

After many RFT rounds, the model may learn to produce outputs that satisfy the verifier's surface-level check (e.g., the exact answer token format) without genuine reasoning improvement. The verifier's pass/fail signal, originally a proxy for correctness, becomes the optimisation target itself. This is Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." It is most severe when the verifier is a learned reward model rather than a ground-truth checker.

DeepSeek-R1 used a rejection sampling stage to construct cold-start reasoning traces before the main reinforcement learning phase. Long reasoning chains were sampled from an intermediate model, filtered for correctness, and used to fine-tune a base model - providing the RL stage with a better starting distribution than a raw pretrained model. This "RFT as RL warm-up" pattern is now common in reasoning model pipelines.

Generator produces candidate outputs. Verifier scores or critiques each candidate. Filter discards low-quality ones and passes the rest back as fine-tuning data for the generator.

The loop compresses capability into the weights without proportionally increasing human labelling effort.

Sample k solutions per problem, execute or symbolically verify each one, discard incorrect solutions, and fine-tune on the correct remainder.

RFT is tractable when a reliable oracle exists (e.g. a maths checker, unit tests). Without an oracle you must rely on the model judging itself, which reintroduces self-consistency limitations.

SL phase: The model critiques and revises its own responses according to natural-language constitutional principles. The final revision becomes supervised training data - no human rater needed.

RLAIF phase: The model ranks response pairs against the constitution. Those AI-generated preference labels train a reward model used for PPO, replacing the human preference labels used in standard RLHF.

Reward hacking / overoptimisation. The generator finds outputs that score well on the proxy reward without being genuinely better. The proxy's blind spots are exploited, and the gap between proxy score and true quality widens as optimisation pressure increases.

This is Goodhart's Law in a statistical setting: once a measure becomes a target, it ceases to be a good measure.

Each filtering pass discards low-scoring outputs, narrowing the distribution of retained examples. Over many iterations, the model is fine-tuned on an increasingly homogeneous set, losing stylistic range and failing on unusual but valid inputs.

Concretely: if the reward signal correlates with response length, the model drifts toward verbosity regardless of quality.

A student model trained on teacher outputs can mimic the surface form of expert responses - correct tone, formatting, instruction-following - without inheriting the teacher's underlying reasoning capability.

Benchmarks measuring surface behaviour look good; benchmarks requiring deep reasoning or factual accuracy do not. The student has learned to "sound like" the teacher, not to reason like one.

A model critiquing its own outputs is bounded by its own blind spots. If the model consistently misunderstands a class of queries, its critiques of responses to those queries will be wrong in the same direction - and fine-tuning on the revised responses entrenches rather than corrects the misunderstanding.

Improvement is only possible where the model already has the capability to distinguish better from worse responses. Constitutional AI partially mitigates this by externalising normative judgement to written principles, but interpretation of those principles is still performed by the same model.

Without persona conditioning, repeated sampling from an LLM produces instructions clustered around the model's training-data prior: English-speaking, tech-literate, Western demographics. Persona prompting inserts a brief identity description into the prompt so each generation call conditions on a different user type, spreading outputs across a wider slice of the real human distribution.

Why it matters: a model fine-tuned on undiverse synthetic data inherits the same coverage gaps and performs poorly for underrepresented user populations.

Text-to-Persona: an LLM is prompted "Who is likely to read or write this text?" on each web document, yielding fine-grained personas grounded in real-world content.

Persona-to-Persona: given a known persona, the model is asked "Who is in a close relationship with this person?" and expanded across six iteration steps to capture demographics with low web footprint (children, rural workers, support staff).

Together, these two methods produced roughly one billion deduplicated personas in the paper.

Without persona conditioning, the k candidates generated per prompt are all drawn from a similar distribution region; rejection sampling then selects the best point in one dense cluster. With persona conditioning, each candidate explores a different persona-relevant region of the space. Rejection sampling now picks the highest-quality representative of each region, yielding a more diverse accepted set.

If a model fine-tuned on persona-conditioned synthetic data is then used to generate the next round of data, the output distribution drifts. Alemohammad et al. (2023) showed that iterative training without fresh real data causes progressive recall (diversity) degradation even if precision (quality) holds briefly -- a condition they call Model Autophagy Disorder. Persona prompting slows but does not prevent this collapse; injecting real data at each generation round is necessary.

Mid-level specificity -- roughly two to four attributes -- works best.

• Too coarse ("a student"): adds no distributional signal; the LLM ignores it.
• Too fine ("a 34-year-old postdoc in computational protein folding with a Bayesian background"): the model has no reliable basis for this identity and fabricates domain details, reducing factual quality.

The sweet spot gives the model enough context to genuinely shift register and topic focus without hallucinating implausible specifics.

1. Stereotype amplification: the LLM's persona-to-output mapping reflects its trained associations. A culturally marked persona (e.g., "a Nigerian entrepreneur") can produce reductive, stereotyped content rather than genuine representational diversity.

2. WEIRD coverage illusion: if the source corpus for persona extraction skews toward Western, Educated, Industrialised, Rich, Democratic populations, the resulting persona pool does too -- regardless of its size. One billion personas drawn from an English-centric web still under-represents large parts of the world.

Persona conditioning adds noise rather than signal when the correct output does not depend on who is asking. A mathematical proof is either valid or it is not; injecting "a primary school teacher" versus "a PhD mathematician" as the persona does not change what constitutes a correct proof, but it does risk lowering the quality of generated solutions. Apply persona prompting selectively, only to tasks where vocabulary, difficulty level, framing, or implicit assumptions meaningfully vary with the user population.

They replaced raw web-scraped training text with roughly one billion tokens of synthetic "textbook-quality" Python exercises generated via GPT-3.5, combined with six billion tokens of carefully filtered real code. The key insight is that data quality (pedagogical clarity, worked examples, concept density) can substitute for data quantity and model scale.

Starting from ~175 hand-written seed tasks, a frozen model generates new instruction-input-output triples by sampling a few seeds as in-context examples and asking the model to produce a novel task. A ROUGE-based filter discards any new task with overlap > 0.7 against the existing pool, and format violations are removed. The generator only needs to be better than random; diversity is enforced by the deduplication step rather than requiring the model to be reliably creative.

The intermediate reasoning trace encodes the process used to reach the answer: decomposition steps, constraint checks, backtracking signals. Final answers discard all of this. A student trained on traces learns a reasoning strategy; a student trained on answers learns only input-output mappings. Orca showed that a 13B model trained on GPT-4 step-by-step explanations substantially outperformed one trained on GPT-4 final answers across complex reasoning benchmarks.

p̃(y|x) ∝ p_θ(y|x) · 1[r(x, y) ≥ τ]

• p_θ(y|x): the generator's probability of output y given input x.
• r(x, y): a scorer (reward model or rule-based verifier) evaluating quality.
• τ: the acceptance threshold.
• 1[...]: the indicator function that zeroes out sub-threshold outputs.

Fine-tuning on samples from p̃ shifts the model toward the accepted region. The distribution is only as useful as the scorer; a flawed scorer systematically biases the student toward whatever the scorer rewards, not what is actually correct.

Supervised phase: The model elicits a harmful output, self-critiques it against a written constitution, revises iteratively, and the fine-tuning data is (original prompt, final revised response). Human input: writing the constitution once; no per-example labelling.

RL phase: The model generates response pairs; a separate "AI feedback" model scores which is preferable per the constitution; this trains a reward model used for PPO. Human input: again only the upfront constitution; no human preference labels on individual pairs.

The scaling advantage is that per-label human annotation is replaced by one-time principle specification.

MAD describes the progressive degradation of quality or diversity when a generative model is repeatedly trained on its own outputs without sufficient fresh real data injected each generation. Each synthetic generation amplifies the current model's biases; rare modes are dropped; the distribution narrows. The trigger is an autophagous (self-consuming) loop where the proportion of real, diverse data falls below what is needed to counteract distributional drift. The fix is to maintain a continuous injection of verified real-world data across training generations.

1. Reward hacking. The scorer is a proxy for quality, not quality itself. A code verifier accepting unit-test passes rewards hard-coded solutions; a verbosity-biased reward model rewards padded answers.

2. Hallucination laundering. A teacher confident but wrong in its chain-of-thought is undetected when the verifier cannot check factual correctness. The student learns to reproduce confident errors.

3. Coverage gaps. The generator is conditioned on its own world model. Rare distributions (minority languages, niche domains) are systematically under-generated; the student fails on exactly the cases synthetic pipelines cannot easily manufacture.

Code can be executed. A unit test either passes or fails, providing a cheap, hard binary signal that requires no learned reward model. Natural language has no equivalent oracle, so filtering must rely on approximations (reward models, self-critique) that carry their own errors. Execution-based filtering is the foundation of every high-quality code synthetic data pipeline.

OSS-Instruct seeds each instruction-generation prompt with a random snippet of real open-source code, then asks the teacher model to construct a related programming task. Because the seed is drawn from the actual diversity of public repositories, the resulting task distribution covers rare library APIs, domain-specific idioms, and language patterns that purely model-generated seeds miss. MagicoderS-CL-7B trained on 75,000 OSS-Instruct examples outperformed ChatGPT on HumanEval+.

Phi-1 combined two data sources: web/StackOverflow content filtered for "textbook quality" (clear explanations, good comments) and GPT-3.5-generated synthetic exercises with worked solutions. The synthetic exercises were explicitly pedagogical, including step-by-step reasoning and edge-case coverage. Signal density per token was high enough to compensate for a corpus orders of magnitude smaller than typical code pretraining data. The result was 50.6% pass@1 on HumanEval at 1.3B parameters.

When a model generates synthetic training data, fine-tunes on it, then generates the next round, each iteration narrows the output distribution. Shumailov et al. (2023) showed this causes "irreversible defects": rare but valid outputs disappear from the model's repertoire. For code, uncommon idioms, less popular languages, and unusual algorithms gradually vanish. The primary mitigation is to ground each generation round in real, human-authored code (e.g., open-source snippets or problem statements), preventing the loop from closing entirely on itself.

1. Depth rewriting - adds constraints, error-handling requirements, or edge cases to an existing instruction, increasing problem difficulty within the same concept.
2. Breadth rewriting - generates a conceptually related but distinct task, expanding the variety of problems covered.

Together they let a small seed set produce a wide spectrum of instruction-code pairs without human authoring, though repeated evolution from a narrow seed can cause the corpus to homogenise over many steps.

Unit tests verify input-output behaviour, not code quality. A solution can pass all tests while being unreadable, algorithmically inefficient (e.g., O(n^2) where O(n log n) is trivial), using deprecated or insecure API calls, or hard-coding special cases that happen to satisfy the test suite but fail on out-of-distribution inputs. Execution-based filtering cannot penalise these properties; a secondary filter (static analysis, a reward model trained on human code-review labels, or mutation testing) is required for higher-quality data.

Verifier leakage occurs when the test suite or seed problems used during rejection sampling overlap with the evaluation benchmark (e.g., HumanEval). Because HumanEval problems are widely known, a teacher model asked to generate "similar coding problems" will often produce near-duplicates. The trained model then scores high not from genuine generalisation but from exposure to problems in the training distribution. Evaluating on held-out benchmarks like EvalPlus or LiveCodeBench (which rotate problems) typically reveals the true lower capability level.

RFT generates many candidate solutions to each problem, executes or checks the final answer programmatically, keeps only the correct ones, and fine-tunes the model on the filtered set. Mathematics is suited to this because final answers (numeric values, expressions) can be verified automatically without human review - the verifier acts as a free, scalable quality filter. Yuan et al. used this to lift LLaMA-7B from 35.9% to 49.3% on GSM8K by pooling correct solutions from multiple model checkpoints.

MetaMath targets problem diversity (not solution diversity) through four operators:

1. Rephrasing - paraphrase the surface form while preserving mathematical content.
2. Self-questioning - decompose a problem into its sub-goal chain.
3. FOBAR - reverse the problem: give the answer, ask for an unknown input.
4. Backward reasoning - start from the target quantity and derive preconditions.

Applied to GSM8K and MATH training splits, this produced MetaMathQA; fine-tuning LLaMA-2-7B on it yielded 66.4% on GSM8K, an 11.5 percentage-point gain over same-size baselines. The intuition mirrors image augmentation: seeing many structural views of the same concept builds a more robust representation.

Outcome supervision gives a single correctness signal for the final answer. A PRM labels each individual reasoning step, allowing training to distinguish solutions that reached the right answer via flawed intermediate steps from genuinely correct chains.

Math-Shepherd automated step-level labelling without human annotation: for each candidate step, the model rolls out multiple completions forward to the end; the step is marked correct if any completion reaches the right answer. This is a noisy but scalable proxy. Lightman et al.'s PRM800K (800k human step annotations) established that process supervision significantly outperforms outcome supervision on the MATH benchmark.

Distribution collapse occurs when a model is repeatedly fine-tuned on its own generated outputs across multiple rounds. Each fine-tuning pass narrows the distribution slightly; over iterations, the model converges to a stereotyped solution style - verbose, formulaic, and brittle on novel problem structures. The diversity of the training signal contracts because no external grounding (human labels, verified novel problems, a stronger teacher) pushes against the narrowing. This is directly analogous to mode collapse in GANs or quality degradation in recursive text self-distillation.

Rejection sampling cannot synthesise solutions that the generator model cannot produce even once across a large sample. Hard problem types - such as combinatorial proofs or multivariable calculus - remain underrepresented because the model's hit rate for them is near zero, so the filter produces almost no training signal for those cases.

Teacher-student distillation partially circumvents this: a stronger teacher (e.g., GPT-4-class) generates high-quality reasoning traces for problem types the student model struggles with; the student is then fine-tuned on those traces. The ceiling is now set by the teacher, not the student - though the teacher's own blind spots propagate downstream.