Learn the stack, end-to-end.

Action Tokenisation and Representation

How continuous robot actions become discrete tokens an autoregressive transformer can emit, from per-dimension binning to frequency-space compression.

Actor-Critic Methods

Actor-critic methods combine a policy network (actor) with a value estimator (critic) to reduce variance in policy gradient updates without the high bias of pure value-based methods.

AlphaFold2 and the Protein-Folding Problem

How a deep-learning system read co-evolution signal out of aligned protein sequences to predict 3D structure at near-experimental accuracy, and what it still cannot do.

AlphaFold3 and Biomolecular Co-Folding

How AlphaFold3 dropped the protein-only structure module for a diffusion head that denoises raw atoms, letting one model co-fold proteins with ligands, nucleic acids, ions, and modified residues, and how the open reimplementations caught up.

Baselines and Variance Reduction

A baseline is a state-dependent function subtracted from the return in policy gradient updates to reduce estimator variance without introducing bias.

Best-of-N and Inference-Time Selection

Best-of-N sampling generates multiple completions from a language model and returns the one ranked highest by a reward model, trading inference compute for quality without updating any weights.

Credit Assignment over Long Generations

Explains why distributing a single scalar reward back across hundreds of generation steps is the central unsolved tension in RL for language models, and surveys the main strategies used to address it.

Deep Q-Networks

DQN combines Q-learning with a deep convolutional network and two stabilisation tricks (experience replay and a target network) to learn Atari-level control policies directly from raw pixels.

Dynamic Programming for RL

Dynamic programming solves RL problems exactly by bootstrapping value estimates across states using the Bellman equations, but only when you have a perfect model of the environment.

Entropy Regularisation

Entropy regularisation adds a bonus term to the RL objective that rewards stochastic policies, improving exploration and preventing premature convergence to deterministic optima.

Evaluating RL-Tuned Models

Standard NLP benchmarks break silently when applied to RL-tuned models because the training objective optimises for a reward signal that can diverge from genuine capability, requiring a distinct evaluation stack to distinguish real improvement from reward gaming.

Exploration in Language-Model RL

Language-model RL training collapses silently when the policy stops generating diverse completions, and standard RL exploration techniques must be reinterpreted to work inside a token-sequence action space.

Few-Shot and In-Context Learning

Learning a task from a handful of worked examples placed in the prompt, with no weight updates, and the surprising evidence about what those examples actually teach.

Generalised Advantage Estimation

GAE introduces a single hyperparameter lambda that smoothly interpolates between high-bias/low-variance TD(0) and low-bias/high-variance Monte Carlo advantage estimates, making policy gradient training substantially more stable.

GRPO: Group Relative Policy Optimisation

GRPO removes the critic network from PPO by estimating baselines from a sampled group of outputs, halving the GPU footprint while delivering competitive reasoning improvements.

Imitation Learning and Diffusion Policies

Why cloning a demonstrator's actions drifts into unseen states, and how generative action models such as diffusion policies and action chunking control the drift.

Injection-Aware Prompt Design

How to structure prompts that consume untrusted input so injection is harder, and why prompt design alone can never make an LLM injection-proof.

Length and Format Reward Hacking

When a reward model assigns higher scores to longer or more structured responses regardless of quality, RL training exploits that signal and the policy degrades into verbose padding and hollow formatting instead of improving reasoning.

Machine Learning for Materials Discovery

How graph neural networks screen millions of candidate crystals for stability and how machine-learning interatomic potentials approximate DFT cheaply enough to simulate them, plus why an in-silico "stable" material is not yet a real one.

Markov Decision Processes

A Markov Decision Process is the formal framework that turns vague "learn from interaction" intuitions into a precise mathematical problem a computer can solve.

Model-Based Reinforcement Learning

Model-based RL learns an explicit dynamics model of the environment and uses it for planning or synthetic data generation, trading model bias for dramatic gains in sample efficiency.

Monte Carlo Methods

Monte Carlo methods estimate value functions by averaging complete episode returns, making them the simplest model-free approach but one that requires episodic tasks and carries high variance.

Multi-Turn and Agentic RL

Multi-turn and agentic RL extends single-response RLHF to sequences of actions across environment steps, requiring credit assignment, trajectory-level rewards, and new training algorithms suited to long-horizon tool-using agents.

Offline RL and the DPO Connection

DPO re-derives the standard KL-regularised RLHF objective and solves it in closed form, turning preference alignment into a supervised classification loss over offline data without ever sampling from the policy during training.

On-Policy vs Off-Policy Learning

On-policy methods learn from data collected by the policy being updated, while off-policy methods learn from data generated by a different behaviour policy, enabling experience reuse but requiring bias corrections.

Pitfalls in ML-for-Science

The failure modes, chiefly data leakage, that make machine-learning results in scientific papers look stronger than they replicate, and the reporting standards proposed to catch them.

Policy Gradients and REINFORCE

Policy gradient methods directly optimise a stochastic policy by estimating the gradient of expected return through sampled trajectories, sidestepping the need to represent a value function over every state-action pair.

Policy Methods vs Value Methods

Policy methods optimise a parameterised policy directly via gradient ascent on expected return, while value methods learn a value function and derive behaviour from it; the distinction shapes sample efficiency, stability, and action-space suitability across the entire RL algorithm landscape.

PPO for Language Models

Proximal Policy Optimisation clips the policy update ratio to prevent destructive gradient steps, making it the workhorse algorithm for RLHF fine-tuning of large language models.

Process vs Outcome Rewards

Process reward models score each reasoning step individually, giving denser training signal than outcome rewards and catching errors before they contaminate a final answer.

Prompt Chaining and Task Decomposition

Splitting a hard task into a pipeline of simpler, individually-checkable prompts so each step can be validated, routed, and debugged on its own.

Prompt Compression

Cutting prompt tokens while holding task performance, via perplexity-based token dropping (LLMLingua) or learned gist tokens, and when prompt caching beats both.

Protein Language Models

How masked-language-model pretraining over amino-acid sequences produces structure and function signal, and how ESMFold trades some accuracy for dropping the MSA search that AlphaFold2 depends on.

Proximal Policy Optimisation

PPO stabilises policy gradient training by clipping the probability ratio between old and new policies, preventing destructively large updates without the computational overhead of second-order methods.

Q-Learning

Q-learning is a model-free, off-policy temporal-difference algorithm that estimates the value of (state, action) pairs and converges to an optimal policy without requiring a model of the environment.

Reasoning RL and R1-Style Training

How DeepSeek-R1 and Kimi k1.5 demonstrated that pure reinforcement learning on verifiable rewards can elicit chain-of-thought reasoning in LLMs without any human-labelled reasoning traces.

Rejection Sampling as RL

Rejection sampling fine-tuning filters a model's own outputs by correctness and trains on the survivors, achieving a policy-improvement step that is mathematically equivalent to one round of RL but without an explicit optimiser loop.

Returns, Discounting, and Episodes

The return is the quantity a reinforcement learning agent actually optimises; discounting controls how far into the future it looks, and whether interactions are episodic or continuing shapes which formulation applies.

Reward Model Calibration and Drift

Reward models trained on human preferences suffer from miscalibration and distribution shift, causing the optimised policy to exploit proxy scores in ways that diverge from actual human intent.

Reward Models as Learned Rewards

A reward model is a classifier trained on human preference comparisons that outputs a scalar score, standing in for the true human utility function during RL fine-tuning of language models.

Reward Over-Optimisation

When a language model is trained too aggressively against a proxy reward model, it learns to exploit the proxy rather than genuinely improve, causing measured reward to climb while true quality declines.

Reward Shaping and Credit Assignment

Reward shaping injects domain knowledge into the reward signal to speed up learning, while credit assignment determines which past actions actually caused a delayed reward.

RLHF as a Reinforcement-Learning Problem

RLHF recasts language model alignment as a policy-optimisation problem where a reward model trained on human comparisons provides the scalar signal that PPO uses to update the language policy.

RLHF: Reward Modeling, PPO, and the DPO Trade-off

How human preference data becomes a reward model, how PPO uses it to fine-tune an LLM, and why DPO often replaces the whole pipeline.

RLVR: RL from Verifiable Rewards

RLVR replaces the trained reward model in RLHF with an automated verifier that checks correctness against a ground-truth answer, producing a clean binary signal that sidesteps reward hacking and scales to tasks like maths and code where answers can be checked programmatically.

RT-2 and Web-Scale Robot Learning

RT-2 co-trains one transformer on internet vision-language data and robot trajectories by encoding actions as text tokens, transferring semantic web knowledge into robotic control.

Sim-to-Real Transfer

Why robot policies are trained in simulation, why they break on real hardware, and how domain randomisation closes the reality gap by making the real world look like one more random draw.

Structured Output Coercion

How to coax reliable JSON, XML, and tabular output from a model using prompting alone, and why that gives you no hard guarantee the way constrained decoding does.

Temporal-Difference Learning

TD learning combines the trial-and-error sampling of Monte Carlo with the bootstrapped updates of dynamic programming to learn value functions online, without waiting for episode ends.

The Bandit Framing of RLHF

RLHF treats a language model as a contextual bandit that receives a single scalar reward per complete response, making full Markov decision process machinery unnecessary but also hiding the dangers of reward over-optimisation.

The Exploration-Exploitation Trade-off

Choosing when to try something new versus repeating what already works is the central tension in reinforcement learning, and getting it wrong kills agent performance regardless of how well the rest of the system is designed.

The Infrastructure of LLM RL

LLM post-training via RL requires four coordinated systems running simultaneously - a policy, a reference model, a reward model, and a value function - and the design choices for each determine both what behaviours emerge and where the training breaks.

The KL-Regularised RL Objective

The KL-regularised RL objective balances reward maximisation against a penalty that keeps the policy close to a reference model, preventing reward hacking while allowing genuine improvement.

Trust-Region Policy Optimisation

TRPO is a policy-gradient algorithm that enforces a KL-divergence constraint on each update, guaranteeing monotonic policy improvement and preventing the catastrophic performance collapses that plague vanilla gradient ascent.

Value Functions and the Bellman Equations

Value functions assign expected cumulative reward to states and state-action pairs; the Bellman equations express these values as self-consistent recursive relationships that underpin every practical RL algorithm.

Vision-Language-Action Models

Turning a pretrained vision-language model into a robot policy that maps camera images plus a language instruction to motor actions, so the robot inherits web-scale semantic knowledge it could never learn from robot data alone.

What is a Large Language Model?

A grounded definition: an LLM is a neural network trained to predict the next token. Everything else — reasoning, tools, agents — is built on top.

Tokens & Tokenization

Models don't see words — they see tokens. Understanding BPE/SentencePiece changes how you reason about context windows, cost, and weird edge cases.

Auditing Synthetic Data

Auditing synthetic training data means applying measurable quality checks, coverage tests, and contamination scans before the data ever touches a training run.

Benchmark Decontamination

Benchmark decontamination is the process of identifying and removing evaluation set examples from a model's pretraining corpus so that reported benchmark scores reflect genuine generalisation rather than memorised answers.

Code Data Curation

A systematic account of how raw source code from the internet is transformed into a deduplicated, filtered, mixed, and decontaminated pretraining corpus for code-focused language models.

Curriculum and Data Ordering

How the sequence and mixture proportions of training batches affect what an LLM learns and when, and why naive i.i.d. sampling often leaves capability on the table.

Data Mixtures and Domain Weighting

Domain weighting determines how much of each data source a model sees during pretraining, and getting this wrong can cost tens of thousands of GPU-hours or silently cripple downstream task performance.

Data Pipelines at Scale

Building a pretraining corpus requires extracting, filtering, deduplicating, and mixing hundreds of billions of tokens from heterogeneous sources while keeping benchmark contamination out.

Data Provenance and Licensing

Data provenance tracks the origin and transformation history of every byte in a training corpus, while licensing determines whether you are legally permitted to use it at all.

Deduplication and Memorisation

Duplicate training examples cause language models to memorise and verbatim-reproduce training text, so removing duplicates before training reduces regurgitation, improves generalisation, and lowers compute waste.

Distillation and Terms-of-Service Constraints

Using a commercial API to generate training data for a competing model almost universally violates the provider's terms of service, and understanding exactly why - and what compliant alternatives exist - is non-negotiable before building any distillation pipeline.

Distilling Reasoning Traces

Reasoning-trace distillation transfers step-by-step chain-of-thought outputs from a large teacher model into fine-tuning data for a smaller student, giving the student reasoning capability it could not develop on final-answer supervision alone.

Diversity Metrics and Collapse Detection

Quantitative metrics for detecting when synthetic training data loses coverage of the original distribution, with practical monitoring strategies to catch collapse before it harms the next model generation.

Document-Level vs Token-Level Deduplication

Document-level deduplication removes whole near-duplicate pages using hashing, while token-level deduplication removes repeated spans within and across documents using suffix arrays, each with distinct cost-quality trade-offs in web-scale corpus construction.

Evol-Instruct

Evol-Instruct is an LLM-driven pipeline that iteratively rewrites seed instructions into progressively harder variants, enabling automated construction of complex instruction-following training data without human labellers.

Exact and Fuzzy Deduplication

How hash-based exact matching and MinHash locality-sensitive hashing together remove the duplicate content that otherwise inflates memorisation and wastes training compute.

Heuristic Quality Filters

Heuristic quality filters are rule-based passes over raw web text that remove boilerplate, malformed documents, and low-information content before any classifier is trained or applied.

Knowledge Distillation from a Teacher Model

Knowledge distillation trains a smaller student model to reproduce a larger teacher's output distribution, enabling compact models with performance well beyond what their size alone would predict.

Language Identification and Filtering

Language identification assigns a language label to each document in a web crawl so pipelines can retain target-language text and discard everything else before quality filtering begins.

MinHash and LSH for Deduplication

MinHash estimates Jaccard similarity between documents in constant memory, and LSH bucketing turns that estimate into a sub-linear nearest-neighbour search, making corpus-scale near-deduplication tractable.

Mixing Synthetic and Human Data

Balancing synthetic and human-authored training examples determines whether a model inherits the strengths of both sources or the worst of each.

Model Collapse from Recursive Training

When a model is trained repeatedly on its own outputs, tail distributions erode and the model progressively forgets rare but important knowledge, eventually producing impoverished, homogenised text.

Multilingual Data Balancing

Multilingual data balancing decides how many tokens from each language land in a pretraining corpus, trading raw web proportions against deliberate upsampling to serve low-resource languages without degrading high-resource ones.

Persona Prompting for Diversity

Persona prompting injects fictional user identities into an LLM's prompt to steer it toward generating training data that spans a broader slice of the real distribution than naive repeated sampling achieves.

PII Detection and Removal

Scrubbing personally identifiable information from web-scale corpora before LLM pretraining reduces memorisation risk and legal exposure, but every detection method trades recall against corpus damage.

Programmatic and Templated Data Generation

Programmatic and templated generation produces synthetic training data through deterministic code, slot-filling templates, and context-free grammars, giving precise control over distribution and format that purely model-driven pipelines cannot match.

Quality Filtering of Synthetic Data

Quality filtering removes low-signal, redundant, or contaminated examples from synthetically generated datasets before fine-tuning, preventing reward hacking and model collapse.

Quality Filtering with Classifiers

Classifier-based quality filtering uses lightweight models trained on curated reference corpora to score and discard low-quality web documents before LLM pretraining.

Rejection Sampling Fine-Tuning and STaR

Rejection sampling fine-tuning and the STaR algorithm are iterative self-improvement techniques that generate synthetic training data by having a model solve problems, keeping only the correct solutions, and re-training on those filtered traces.

Self-Instruct

Self-Instruct is a bootstrapping pipeline that uses a language model's own outputs to generate large-scale instruction-following training data with minimal human annotation.

Self-Play and Self-Improvement

Self-play loops use a model to generate, critique, and filter its own training data, compounding capability without proportionally scaling human annotation effort.

Synthetic Data for Code

Generating synthetic code training data via instruction synthesis, distillation, and execution-based filtering lets small models punch well above their weight, but only when a reliable verifier anchors the loop.

Synthetic Data for Mathematics

Synthetic math data pipelines use teacher models, rejection sampling, and question rewriting to bootstrap training corpora far larger than any human-labelled set, but the quality ceiling is set by the generator's own reasoning ability.

Synthetic Preference Data

Synthetic preference data replaces expensive human comparison labels by having a language model judge which of two candidate responses is better, enabling scalable RLHF-style alignment without a large human labelling workforce.

Textbook-Quality Synthetic Data

How to generate training data that teaches models to reason rather than memorise, using instruction synthesis, distillation, rejection sampling, and constitutional loops, along with the collapse risks that follow.

Text Extraction and Boilerplate Removal

Converting raw HTML from web crawls into clean, main-content text is a lossy signal-recovery problem, and the choices made here propagate irreversibly through every downstream filtering and training stage.

The Attention Mechanism

Attention lets every token decide which other tokens to look at. It's the core operation that made transformers replace RNNs.

The Constitutional AI Data Loop

Constitutional AI replaces most human preference labels with a self-critique-and-revise loop guided by a written list of principles, producing both supervised fine-tuning data and AI-labelled preference pairs that train a reward model.

Toxicity Filtering of Pretraining Data

Toxicity filtering removes hate speech, obscene language, and harmful content from web-crawled corpora before LLM pretraining, using blocklists, classifiers, and API-based scoring, each with distinct recall/precision trade-offs and equity side effects.

Training a Tokeniser on a Corpus

Training a tokeniser on your pretraining corpus, rather than borrowing one designed for another model, directly controls sequence length, vocabulary coverage, and how the model sees numbers, code, and non-English text.

Vocabulary Size Trade-offs

Choosing a tokeniser vocabulary size forces a three-way tension between sequence length, embedding table memory, and coverage of rare or multilingual text.

Web-Scale Corpus Construction

Building a pretraining corpus at web scale requires five tightly coupled stages - extraction, quality filtering, deduplication, source mixing, and decontamination - each of which silently determines what a model can and cannot know.

Why Synthetic Data

Synthetic data lets practitioners generate arbitrarily large, precisely labelled training sets without manual annotation, but only when the generator is accurate enough not to amplify its own errors.

Pretraining vs Instruction Tuning

How a base model becomes a chatbot. Three stages — pretraining, supervised fine-tuning, RLHF — each doing something specific.

Adapter Modules

Adapter modules insert small trainable bottleneck layers into a frozen pretrained transformer, achieving near-full fine-tuning performance while updating fewer than 4% of the model's parameters.

Autotuning GPU Kernels

Autotuning systematically searches a discrete configuration space of tile sizes, warp counts, and pipeline stages to find the fastest kernel for a given GPU and problem shape, replacing manual heuristics with empirical benchmarking.

Building a Preference Dataset

A preference dataset pairs model outputs and records which one a human (or AI judge) preferred, providing the training signal that separates a helpful assistant from a raw base model.

Catastrophic Forgetting in Fine-Tuning

Full fine-tuning on a narrow task silently destroys general capabilities baked in during pre-training, and understanding why this happens is the prerequisite for choosing any mitigation strategy.

Chat Templates and Special Tokens

Chat templates are Jinja2 strings stored in a tokeniser that convert structured message lists into the exact token sequences a fine-tuned model was trained to process.

Choosing LoRA Rank and Alpha

LoRA rank r controls how much task-specific capacity the adapter has, and alpha controls the scaling of that update; choosing them poorly wastes parameters or destabilises training.

Collective Communication Primitives

The six core multi-GPU communication patterns (broadcast, reduce, all-reduce, all-gather, reduce-scatter, all-to-all) determine whether a distributed training job spends most of its time computing or waiting on the wire.

Composing and Stacking Adapters

Multiple trained adapters can be combined sequentially, by weighted sum, or through attention-based gating to build new capabilities without retraining the base model.

Compute-Bound vs Memory-Bound Kernels

A kernel's performance ceiling is determined by whether FLOPs or memory bandwidth runs out first, and misidentifying this wastes orders-of-magnitude optimisation effort.

Custom Kernels for Mixture-of-Experts

MoE models break the dense-GEMM assumption that GPU libraries are optimised for, so efficient inference requires custom grouped-GEMM and block-sparse kernels that handle variable-length expert batches without padding or token dropping.

DoRA: Weight-Decomposed LoRA

DoRA decomposes pre-trained weights into magnitude and direction components, then applies LoRA exclusively to the directional part, closing most of the accuracy gap between LoRA and full fine-tuning without adding inference overhead.

DPO in Practice

DPO eliminates the separate reward model and RL loop of classic RLHF by reparameterising the reward directly into a classification loss over preferred and rejected response pairs.

Evaluating an Aligned Model

Evaluating an aligned model requires measuring three partially competing properties simultaneously: helpfulness, harmlessness, and honesty, and every method for doing so introduces its own systematic biases.

Floating-Point Formats for ML

ML accelerators expose a menu of floating-point formats that trade numerical range and precision for throughput and memory bandwidth; choosing the wrong one silently degrades accuracy or leaves peak FLOPS on the table.

Graph Capture and CUDA Graphs

CUDA Graphs record a sequence of GPU operations as a reusable execution graph, eliminating per-kernel CPU launch overhead and enabling significant throughput gains for workloads with static shapes and control flow.

Hardware Cost of Mixture-of-Experts

Sparse MoE models reduce FLOPs per token but introduce all-to-all communication, load-imbalance penalties, and memory pressure that can erase those savings unless the system is carefully co-designed.

HBM Bandwidth and Capacity

High Bandwidth Memory sets a hard ceiling on how fast a GPU can feed its compute units, and most LLM operations live squarely against that ceiling.

IA3 and Scaling-Vector Methods

IA3 fine-tunes a transformer by learning one scaling vector per targeted activation stream, achieving roughly 0.01% trainable parameters while matching full fine-tuning accuracy on several benchmarks.

InfiniBand and Inter-Node Networking

InfiniBand provides low-latency, high-bandwidth RDMA links between GPU nodes, and understanding its topology and collective communication patterns is essential for diagnosing and eliminating the network bottleneck in large-scale training.

IPO and the Overfitting Fix

IPO replaces DPO's sigmoid loss with a squared identity transform, eliminating the theoretical overfitting guarantee that breaks when preference data is finite and deterministic.

Kernel Fusion

Kernel fusion eliminates redundant memory round-trips by merging multiple GPU operations into a single kernel launch, turning memory-bandwidth bottlenecks into throughput wins.

KTO: Unpaired Preference Learning

KTO aligns language models using only binary good/bad labels per response, avoiding the paired (chosen, rejected) format that makes preference data expensive and brittle to collect.

KV-Cache Memory and Bandwidth

The key-value cache trades GPU memory capacity for inference speed, and understanding how that trade interacts with memory bandwidth is what separates fast serving systems from slow ones.

Length Bias and Verbosity Control

Reward models trained on human preference data systematically score longer outputs higher regardless of quality, causing RLHF-trained models to inflate response length rather than improve content.

LoRA for Long-Context Adaptation

Extending a model's context window via LoRA requires coordinating low-rank weight updates with position-encoding rescaling, and ignoring either side reliably degrades performance on long sequences.

LoRA: Low-Rank Adaptation

LoRA freezes a pretrained model's weights and inserts trainable low-rank matrix pairs into each target layer, cutting trainable parameters by up to 10,000x with no added inference latency.

LoRA Training Pitfalls

LoRA's low-rank approximation introduces subtle failure modes around rank selection, learning rate asymmetry, and target-module coverage that can silently degrade fine-tuned model quality.

LoRA vs Full Fine-Tuning

LoRA constrains weight updates to low-rank matrices, cutting trainable parameters by orders of magnitude while matching full fine-tuning quality on most tasks, but that constraint is also the source of its failure modes.

Memory Coalescing

Memory coalescing is the hardware mechanism by which a GPU groups multiple thread memory requests into a single wide transaction, and writing kernels that exploit it is often the single largest lever on throughput.

Merging LoRA into Base Weights

After LoRA training, the low-rank adapter matrices can be folded directly into the frozen base weights, eliminating inference overhead and adapter management complexity while producing a standard dense model.

Mixed-Precision Kernels

Mixed-precision kernels reduce memory bandwidth and arithmetic cost by storing and computing in lower-precision formats while selectively preserving full precision where numerical stability demands it.

Model Merging: Linear and SLERP

Linear and SLERP merging combine the weight tensors of separately fine-tuned models into a single deployable checkpoint, trading off alignment and capability at zero inference cost.

Model Soups

Model soups average the weights of multiple independently fine-tuned checkpoints to produce a single model that outperforms any individual checkpoint without increasing inference cost.

Multi-Objective Alignment

Multi-objective alignment trains a single language model to satisfy several competing criteria simultaneously by navigating the Pareto front of reward trade-offs rather than collapsing them into one scalar.

NF4 and Double Quantisation

NF4 is a 4-bit data type matched to the normal distribution of pretrained weights, and double quantisation further compresses the quantisation constants themselves, together enabling 65B-parameter models to fine-tune on a single 48 GB GPU via QLoRA.

NVLink and Intra-Node Interconnect

NVLink is NVIDIA's proprietary GPU-to-GPU interconnect that delivers up to 900 GB/s aggregate bandwidth on H100, replacing PCIe as the bottleneck in multi-GPU training by making all-reduce and tensor parallelism far cheaper.

Occupancy and Latency Hiding

GPU occupancy measures how many warps are resident on a streaming multiprocessor relative to its hardware maximum, and high occupancy is the primary mechanism by which the GPU hides memory and arithmetic latency to sustain throughput.

Online vs Offline Preference Optimisation

Offline preference optimisation trains on a fixed dataset of ranked responses, while online methods continuously sample from the current policy, and that single difference has substantial consequences for distribution coverage, reward hacking risk, and final alignment quality.

Operator Lowering and IRs

Operator lowering is the process of progressively translating high-level tensor operations through a sequence of intermediate representations until hardware-executable instructions are produced.

ORPO and Reference-Free Alignment

ORPO collapses supervised fine-tuning and preference alignment into a single training phase by appending a log-odds-ratio penalty directly to the NLL loss, removing the need for a reference model.

Paged Attention as a Memory Manager

PagedAttention borrows the OS virtual-memory paging model to eliminate KV-cache fragmentation, letting a single GPU serve far more concurrent requests than contiguous allocation allows.

Power, Thermals, and Clock Throttling

GPU accelerators operate under hard power and thermal budgets that silently reduce clock speeds mid-workload, making sustained throughput lower than peak spec sheets advertise.

PPO for RLHF in Practice

A concrete walkthrough of how Proximal Policy Optimisation is wired into the RLHF pipeline, covering the four-model setup, the clipped objective, KL penalty shaping, and the failure modes that kill real training runs.

Prefill vs Decode

LLM inference splits into two hardware-distinct phases - a compute-bound prefill that processes all prompt tokens in parallel, and a memory-bandwidth-bound decode that generates tokens one at a time, each with fundamentally different bottlenecks on the same GPU.

Prefix Tuning

Prefix tuning freezes all pretrained model weights and instead optimises a small set of continuous, task-specific vectors prepended to every layer's key-value cache, achieving within a few points of full fine-tuning while training roughly 0.1% of the original parameters.

Profiling GPU Workloads

Profiling a GPU workload means measuring where time and memory bandwidth actually go, so that optimisation effort lands on the real bottleneck rather than a guess.

Prompt Engineering — Foundations

Prompting is a skill, not a hack. Five techniques that work across models: clear role, structured output, examples, decomposition, and verification.

Prompt Tuning with Soft Prompts

Soft-prompt tuning prepends a small set of learnable continuous vectors to the input of a frozen language model, achieving full-fine-tuning parity at billion-parameter scale while touching less than 0.1% of model weights.

QLoRA: 4-bit Base with LoRA

QLoRA reduces the GPU memory needed to fine-tune a 65-billion-parameter model from hundreds of gigabytes to a single 48 GB card by storing the frozen base model in 4-bit precision and training only small LoRA adapters at full 16-bit precision.

Quantised GEMM Kernels

Quantised GEMM kernels replace 16-bit or 32-bit matrix multiplications with 8-bit or 4-bit integer arithmetic, cutting memory bandwidth and compute cost while preserving model accuracy through careful scaling and outlier handling.

Reading an Accelerator Datasheet

A datasheet number means nothing without the four unit-aware ratios that reveal whether your workload will actually be compute-bound or memory-bound on that chip.

Reward Hacking in Alignment

Reward hacking occurs when a model maximises its training reward signal through behaviours that violate the designer's intent, undermining alignment despite high measured scores.

Reward Modelling in Practice

A reward model is a learned surrogate for human preference that RLHF uses to provide a differentiable training signal, and its quality determines how aligned the final policy is.

RLAIF and Constitutional Feedback

RLAIF replaces human preference labels with an AI judge, and Constitutional AI extends this by encoding a written list of principles so the model critiques and revises its own outputs before any RL training begins.

Roofline-Guided Kernel Optimisation

The roofline model maps a kernel's arithmetic intensity against hardware ceilings to diagnose whether compute or memory bandwidth is the binding constraint, and directs every subsequent optimisation decision.

Serving Many LoRA Adapters

How specialised inference systems batch requests across hundreds of distinct LoRA adapters without duplicating the base model weights on GPU.

Shared Memory and Tiling

Shared memory is a programmer-controlled on-chip SRAM that lets a thread block reuse data without re-fetching it from global memory, and tiling is the technique that makes that reuse systematic.

Supervised Fine-Tuning for Instructions

Supervised fine-tuning on curated instruction-response pairs is the first step that transforms a raw pretrained language model into a model that reliably follows human instructions.

Tensor Cores and Matrix Engines

Tensor Cores are specialised matrix-multiply-accumulate units on modern GPUs that deliver peak FLOP/s only when operand shapes and numeric formats are chosen correctly.

The CUDA Programming Model

CUDA organises GPU execution into a three-level hierarchy of grids, blocks, and threads, and every performance decision traces back to how well that hierarchy is exploited.

The FLOPs of a Transformer Forward Pass

A systematic derivation of how many floating-point operations a single transformer forward pass costs, and why that number dictates hardware choice, batch strategy, and scaling decisions.

The GPU Execution Model

GPUs execute thousands of threads in lockstep groups called warps; understanding that hierarchy and where threads stall is the single most important mental model for writing fast GPU code.

The GPU Memory Hierarchy

A GPU's memory is a multi-tier hierarchy where bandwidth drops and latency rises by orders of magnitude as you move outward from registers to HBM, and the speed of your kernel is almost always determined by which tier bottlenecks it.

The Intrinsic-Dimension Hypothesis

Pre-trained language models can be fine-tuned in a surprisingly low-dimensional subspace of their parameter space, and measuring that dimension explains why parameter-efficient methods work at all.

The KL Penalty and Reference Model

The KL penalty constrains a fine-tuned language model to stay statistically close to its pre-trained reference, preventing reward hacking while preserving the capabilities built during pretraining.

The Memory Maths of Fine-Tuning

Fine-tuning a 7B-parameter model with full gradients and Adam state consumes roughly 112 GB of GPU memory; parameter-efficient methods cut that by an order of magnitude by training only a tiny fraction of weights.

The Memory Wall and Arithmetic Intensity

Arithmetic intensity determines whether a GPU kernel is memory-bound or compute-bound, and almost every LLM inference operation sits on the wrong side of that line.

The Post-Training Pipeline

A structured walkthrough of the four-stage process that converts a raw pretrained language model into a deployable assistant, from supervised fine-tuning through reward modelling, RLHF, and preference-optimisation alternatives.

The Roofline Model

The Roofline Model bounds attainable hardware performance using two ceilings - peak compute throughput and peak memory bandwidth - letting you diagnose whether a kernel wastes silicon or wasits time waiting for data.

TIES and DARE Merging

TIES and DARE are two parameter-space merging algorithms that resolve weight interference when combining multiple fine-tuned models into one, avoiding retraining entirely.

torch.compile and TorchInductor

torch.compile traces PyTorch graphs at runtime via TorchDynamo, then lowers them through TorchInductor to fused Triton or C++ kernels, delivering 20-36% throughput gains with no model rewrites.

TPU Systolic Arrays

A systolic array is a grid of multiply-accumulate units wired to pass partial sums directly between neighbours, letting Google's TPU sustain 92 TOPS on matrix multiplication without repeatedly hitting off-chip memory.

Triton: Python-Level GPU Kernels

Triton lets you write GPU kernels in Python by operating on tiles of data rather than individual threads, and its compiler handles shared-memory management, coalescing, and vectorisation automatically.

What a CUDA Kernel Is

A CUDA kernel is a C++ function that runs simultaneously on thousands of GPU threads, each identified by a coordinate in a structured grid, and understanding this execution model is the prerequisite for reasoning about throughput in any deep-learning workload.

When Not to Write a Custom Kernel

Writing a CUDA kernel is expensive to maintain and easy to get wrong; this concept maps the decision boundary between writing one and leaning on existing compilers and libraries.

Which Layers to Adapt

Choosing which transformer weight matrices to inject LoRA or adapter modules into determines both the parameter budget and downstream task quality, and the right choice is not self-evident.

Why FlashAttention Is a Kernel Story

FlashAttention achieves its speedups not by reducing FLOPs but by restructuring the attention computation into a single tiled CUDA kernel that fits working data in on-chip SRAM, eliminating the dominant cost of round-tripping through GPU HBM.

Why GEMMs Dominate

Almost every compute-heavy operation in a neural network reduces to a matrix multiply, which is why hardware and compilers optimise almost exclusively for GEMM throughput.

Why Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning methods adapt large pretrained models to new tasks by training only a small fraction of parameters, making customisation practical without the compute and storage costs of full fine-tuning.

Writing a Fused Softmax

A fused softmax kernel collapses three separate memory-bound passes over a matrix row into one, cutting HBM traffic by roughly 4x and turning a memory-bound operation into a compute-limited one.

XLA and Just-In-Time Compilation

XLA compiles a whole computation graph into fused, hardware-specific kernels at runtime, trading a one-time compilation cost for sustained throughput gains across GPUs and TPUs.

Attention Failures in TTS

Attention-based TTS systems fail in predictable ways - word skipping, repetition, and unstable alignment - and understanding the mechanics behind each failure mode is essential for building reliable speech synthesis pipelines.

Audio Features and Spectrograms

Raw audio waveforms are rarely fed directly to speech models; this concept explains how and why they are first converted into spectrogram-based representations that compress perceptual information into a learnable 2-D grid.

Bark and Fully Generative Audio

Bark is a transformer-based model that generates speech, music, and nonverbal audio from text by autoregressively predicting discrete audio codec tokens, without any phoneme pipeline or continuous acoustic model.

Beam Search Decoding in ASR

Beam search decoding navigates the exponentially large label sequence space in ASR by maintaining a fixed-width frontier of the most probable partial hypotheses at each step, making it the default inference strategy for CTC, RNN-T, and attention-based models.

Causal and Chunked Attention for Streaming

Streaming ASR requires attention mechanisms that never look at future audio; causal masking and chunked attention are the two principal techniques, each trading latency against accuracy in different ways.

Contrastive Vision-Language: CLIP

How a 400M image-text contrastive objective produced a shared embedding space that does zero-shot classification, retrieval, and grounding without any task-specific labels.

CTC: Connectionist Temporal Classification

CTC is a training objective that lets a neural network learn to align variable-length audio to text without any hand-labelled frame-level annotations.

Diffusion Models

How learning to invert a noise process became the dominant generative recipe for images, video, and audio, and why Flow Matching and DiTs are reshaping the recipe in 2024.

Diffusion Models for Speech

Diffusion models iteratively denoise random Gaussian noise into speech waveforms or mel-spectrograms, achieving sample quality that matches autoregressive vocoders at a fraction of the sequential compute cost.

Duration Modelling and Alignment

Duration modelling assigns how many audio frames each phoneme occupies; alignment is the mechanism that learns or infers that mapping from text-audio pairs without manual annotation.

Endpointing and Voice Activity Detection

Endpointing and voice activity detection are the mechanisms that decide when a user has finished speaking, directly controlling the latency and correctness of every streaming ASR system.

Evaluating TTS with MOS

Mean Opinion Score is the field's primary yardstick for TTS naturalness, but its reliability hinges on listener pool design, anchoring, and context choices that most papers under-report.

FastSpeech and Non-Autoregressive TTS

FastSpeech eliminates the sequential mel-frame dependency of autoregressive TTS by predicting all frames in parallel, trading model simplicity for a duration predictor and a length regulator.

Griffin-Lim and Classical Vocoders

Griffin-Lim is an iterative phase-recovery algorithm that converts a magnitude spectrogram back into a waveform without any learned parameters, and understanding it clarifies exactly what neural vocoders had to replace and why.

HiFi-GAN and Neural Vocoders

Neural vocoders convert acoustic feature representations into raw audio waveforms, and HiFi-GAN achieves near-human quality at 167x real-time speed using a multi-scale, multi-period GAN architecture.

HuBERT and Discrete Audio Units

HuBERT pre-trains a speech encoder by predicting offline k-means cluster labels for masked audio frames, producing discrete unit sequences that rival phoneme transcriptions without any text supervision.

Language-Model Fusion

Language-model fusion techniques inject text-only knowledge into end-to-end ASR models at inference time or training time, and the correct method depends on how much implicit language bias the acoustic model has already absorbed.

Listen, Attend and Spell

LAS is a purely sequence-to-sequence ASR model that replaces HMMs, CTC, and explicit pronunciation dictionaries with a pyramidal RNN encoder and an attention-based character decoder trained end-to-end.

Multimodal LLMs: LLaVA, Flamingo, GPT-4V

The vision-encoder-plus-projector-plus-LLM recipe that dominates open multimodal models, why Flamingo's perceiver design still matters for video, and what native-multimodal frontier models do differently.

Neural Audio Codecs

Neural audio codecs compress waveforms into discrete token sequences using learned vector quantisation, enabling language models to generate speech token-by-token.

Prosody and Style Control

How TTS systems encode and manipulate pitch, duration, energy, and speaking style so that synthesised speech sounds intended rather than merely intelligible.

Residual Vector Quantisation

Residual vector quantisation stacks multiple codebooks to approximate a continuous audio embedding with increasingly fine-grained corrections, making it the compression backbone of modern neural audio codecs.

Robustness to Noise and Accents

How modern ASR systems are trained and adapted to handle environmental noise, channel distortions, and speaker accent variability without collapsing to near-zero accuracy.

Segment Anything (SAM) and Dense Prediction

How promptable segmentation became a foundation-model task, what SAM's encoder-decoder split was designed for, and where it still loses to specialist models.

Speaker Diarisation

Speaker diarisation segments an audio recording into speaker-homogeneous regions and assigns each region a speaker identity, answering the question "who spoke when" without necessarily transcribing what was said.

Streaming TTS and Latency

Streaming TTS pipelines generate and deliver audio incrementally to cut time-to-first-audio from several seconds to under 300 ms, but doing so imposes hard trade-offs on chunk size, model architecture, and prosodic coherence.

Streaming vs Offline ASR

Streaming ASR emits partial transcripts incrementally as audio arrives, trading access to future context for low latency, while offline ASR processes the full utterance and consistently achieves lower word error rates.

Tacotron 2

Tacotron 2 is a two-stage neural TTS pipeline that converts text to mel spectrograms with a sequence-to-sequence model, then synthesises raw audio with a conditioned WaveNet vocoder, achieving near-human MOS scores.

Text Normalisation and Phonemisation

Text normalisation converts raw written text into a speakable form, and phonemisation maps those words to phoneme sequences; together they determine what a TTS system says before any audio is generated.

The Acoustic Model and Vocoder Split

Modern neural TTS splits the problem into two specialised sub-networks - an acoustic model that maps text to a compact spectral representation, and a vocoder that reconstructs a full audio waveform from that representation.

The ASR Problem and Pipeline

Automatic speech recognition converts a raw audio waveform into a word sequence by solving an alignment problem that classical NLP never had to face.

The Conformer Architecture

The Conformer interleaves convolution and multi-head self-attention inside each encoder block to capture both fine-grained local acoustic patterns and long-range sequence dependencies, achieving state-of-the-art ASR accuracy on LibriSpeech.

The CTC Blank Token and Alignment

The CTC blank token is a special output symbol that lets a neural network emit one label per time-step without needing a hand-crafted alignment between audio frames and characters.

The Mel-Spectrogram Interface

The mel spectrogram is the agreed-upon intermediate representation that decouples acoustic modelling from waveform generation in modern TTS pipelines.

The RNN-Transducer

The RNN-Transducer is a fully neural, streaming-capable sequence transduction model that replaces CTC's conditional independence assumption with a learned label-context network, enabling accurate on-device speech recognition.

The TTS Problem and Pipeline

Text-to-speech converts a string of characters into a waveform through a chain of normalisation, acoustic modelling, and synthesis stages, each introducing its own failure modes.

VALL-E: TTS as Token Language Modelling

VALL-E reformulates text-to-speech as a conditional language modelling problem over discrete audio codec tokens, enabling zero-shot voice cloning from a three-second recording by treating acoustic context the same way GPT treats a few-shot text prompt.

Video, Audio, and Any-to-Any Models

How Whisper, V-JEPA, Sora-class video generators, MusicGen, and unified any-to-any models extend the multimodal stack beyond static images.

Vision Transformers (ViT)

How treating an image as a sequence of patches let pure transformers beat CNNs once data crossed the 300M-image mark, and what the architecture gave up to get there.

wav2vec 2.0 and Self-Supervised Audio

wav2vec 2.0 learns speech representations from raw audio without transcripts by masking latent features and solving a contrastive task over learned discrete units, then fine-tunes on as little as ten minutes of labelled speech to reach competitive word error rates.

WaveNet

WaveNet is a fully autoregressive convolutional neural network that models raw audio waveforms one sample at a time, achieving near-human speech quality at the cost of extremely slow sequential generation.

Whisper's Multitask Decoder

Whisper conditions a single sequence-to-sequence decoder on a prefix of special tokens that specify language, task, and timestamp behaviour, allowing one model to handle transcription, translation, and language identification without any task-specific heads.

Whisper: Weakly-Supervised ASR

Whisper trains a sequence-to-sequence Transformer on 680,000 hours of weakly-supervised internet audio to achieve robust multilingual speech recognition without task-specific fine-tuning.

Why Sequence Length Is Hard in Audio

Audio produces roughly 100 frames per second of speech, making sequence lengths 10-50x longer than equivalent text, and this mismatch drives almost every architectural and training decision in modern ASR.

Word Error Rate and ASR Evaluation

Word Error Rate measures ASR accuracy as the minimum edit distance between a hypothesis and a reference transcript, normalised by reference length, but its apparent simplicity hides a nest of normalisation choices that make numbers across papers routinely incomparable.

Zero-Shot Voice Cloning

Zero-shot voice cloning synthesises speech in the voice of an unseen speaker from a short reference recording, without any fine-tuning at inference time.