Multi-Head Latent Attention: How DeepSeek Compressed the KV Cache Without Losing Quality

In May 2024, a research team in Hangzhou published a paper claiming their new attention mechanism was smaller and better than the standard alternative at the same time. That is an unusual claim in machine learning, where memory, speed, and quality are normally traded off against each other rather than improved together. The mechanism was Multi-Head Latent Attention (MLA), and the model was DeepSeek-V2: 236 billion total parameters, 21 billion active per token, and a key-value cache 93.3% smaller than the dense 67B model it replaced (DeepSeek-AI, 2024, DeepSeek-V2, arXiv:2405.04434).

Why this matters: Every production LLM serving system is bottlenecked less by raw compute than by how many bytes it must move from GPU memory for every generated token. The key-value cache, not the model weights, is usually what limits how many users a single GPU can serve concurrently at long context lengths. MLA attacks that bottleneck directly, and its adoption by DeepSeek-V3, Kimi K2, and (in descendant form) GLM-5 has made it one of the most consequential architecture choices in open-weight LLMs since Grouped-Query Attention.

TL;DR

• Standard multi-head attention (MHA) caches a full key and value vector for every attention head, at every layer, for every token. That cache grows linearly with sequence length and head count, and at long context it dwarfs the model weights themselves in memory footprint.
• Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) shrink the cache by sharing key/value heads across query heads, but they pay for it with a measurable quality gap versus full MHA.
• Multi-Head Latent Attention instead compresses the keys and values into a single low-rank latent vector per token, then reconstructs per-head keys and values from that vector on demand. Only the latent vector needs to be cached.
• DeepSeek-V2 reports a 93.3% KV cache reduction and a 5.76x increase in maximum generation throughput relative to its dense 67B predecessor, alongside a 42.5% cut in training cost (DeepSeek-AI, 2024, arXiv:2405.04434).
• Compressing keys and values breaks standard RoPE, because rotary position encoding needs to act on the pre-compression content. DeepSeek's fix, decoupled RoPE, carves out a small dedicated slice of each query/key vector purely for position information and leaves the rest of the vector compressible.
• MLA is not free: it trades memory bandwidth for extra matrix multiplications, and independent open-source re-implementations of the reference DeepSeek-V2 code found that naive KV caching is sometimes faster in practice than the "textbook" compressed caching, because of that added compute (Banatt, On MLA).
• Adoption is not universal. MiniMax explicitly rejected MLA-style and hybrid attention for its M2 model after finding no efficient variant matched full attention quality on long-context reasoning and agentic tasks (MiniMax, 2026, arXiv:2605.26494), while Kimi K2 and GLM-5's descendant architecture lean the other way.

At a Glance

flowchart LR
    Token["Input token"] --> Compress["Compress K,V into latent vector"]
    Compress --> Store["Store latent in KV cache"]
    Store --> Retrieve["Retrieve latent on next step"]
    Retrieve --> Expand["Expand to per-head K and V"]
    Expand --> Score["Attention scoring and output"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff

    class Token blue
    class Compress,Store,Retrieve amber
    class Expand,Score teal

The entire idea fits in that loop: compress before caching, expand only when the math needs the full-size vectors. Everything else in this article is detail on how to make that loop cheap, accurate, and compatible with position encoding.

[IMAGE: Side-by-side schematic comparing MHA's per-head K/V storage boxes against MLA's single shared latent-vector box, with arrows showing the up-projection step that reconstructs per-head K and V]

Before MLA: A Short History of Shrinking the Cache

Transformers generate text one token at a time, and at every step the model must attend over every previous token's key and value vectors (Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762). Recomputing those vectors from scratch at each step would be wasteful, since past tokens' keys and values never change once computed. The standard fix, KV caching, stores them instead. That works well until the cache itself becomes the bottleneck: for a 128-head, 128-dimension-per-head model, every token needs roughly 64 KB of cache per layer in 16-bit precision, and that number multiplies across dozens of layers and every concurrent user a server is handling.

Noam Shazeer's 2019 paper attacked this directly with Multi-Query Attention, in which every query head shares a single key head and a single value head instead of having its own (Shazeer, 2019, arXiv:1911.02150). The cache shrinks by roughly the number of heads, and decoding gets dramatically faster because far less data needs to move through memory bandwidth. The tradeoff is real: forcing every head to read from identical keys and values narrows what the attention mechanism can represent, and MQA models measurably lose quality relative to full MHA.

Grouped-Query Attention split the difference in 2023. Instead of one shared KV head for the whole model, GQA divides query heads into groups, and each group shares one KV head (Ainslie et al., 2023, arXiv:2305.13245). With enough groups, quality recovers close to full MHA; with fewer groups, the cache shrinks close to MQA levels. GQA became the default for most open-weight LLMs through 2023 and 2024 precisely because it is a tunable dial between the two extremes, not because either extreme was ideal.

DeepSeek's 2024 contribution reframed the problem instead of tuning the same dial further. Rather than reducing the number of distinct KV heads, MLA keeps every head fully expressive and instead compresses what gets stored. The cached object is a low-rank latent vector, not a set of per-head keys and values, and the per-head vectors are reconstructed from that latent representation only when the attention computation actually needs them.

timeline
    title Evolution of Attention for Inference Efficiency
    2017 : Transformer introduces full multi-head attention (Vaswani et al.)
    2019 : Multi-Query Attention shares one KV head across all queries (Shazeer)
    2023 : Grouped-Query Attention tunes a dial between MHA and MQA (Ainslie et al.)
    2024 : Multi-Head Latent Attention compresses KV into a latent vector (DeepSeek-V2)
    2024 : DeepSeek-V3 scales MLA to a 671B parameter Mixture-of-Experts model
    2025 : TransMLA converts existing GQA checkpoints into MLA post hoc
    2026 : Kimi K2 and GLM-5's descendant architecture adopt MLA and MLA-derived sparse attention at trillion-parameter scale, while MiniMax M2 opts for full attention instead

[IMAGE: Timeline infographic of attention-efficiency milestones from the 2017 Transformer to 2026 trillion-parameter deployments, with cache-size-per-token annotated at each milestone]

How MLA Actually Works

The core trick: compress before you cache, expand only when you compute

Standard attention computes queries, keys, and values from the hidden state \(h_t\) at position \(t\) using three separate projection matrices. MLA replaces the key/value projection with two smaller matrices chained together. First, a down-projection compresses \(h_t\) into a latent vector \(c_t^{KV}\) with a much smaller dimension than the concatenated keys and values would normally require. Second, an up-projection expands that latent vector back out into the full-size keys and values that attention needs:

\[c_t^{KV} = W^{DKV} h_t, \qquad k_t = W^{UK} c_t^{KV}, \qquad v_t = W^{UV} c_t^{KV}\]

In DeepSeek-V2, the model dimension supports 128 attention heads of 128 dimensions each, so a naive cache would need to store $2 \times 128 \times 128 = 32{,}768$ numbers per token per layer. The compressed latent vector \(c_t^{KV}\), by contrast, has only 512 dimensions. Because \(k_t\) and \(v_t\) can always be reconstructed from \(c_t^{KV}\) on demand, only the 512-dimensional latent needs to sit in the cache; the up-projection happens fresh at attention time. Queries get the same treatment through a separate, larger compression dimension (1{,}536 in DeepSeek-V2), though the query-side compression saves activation memory during training rather than KV cache at inference, since queries are never cached across steps.

This is matrix rank factorization applied directly to the attention projections: a \((d, 2d)\) matrix can be replaced by two smaller matrices, \((d, r)\) and \((r, 2d)\), whenever \(r\) is small enough that \(dr + 2dr < 2d^2\), which holds for essentially any useful compression ratio (Banatt, On MLA). The bet is that the keys and values a well-trained transformer actually needs live close to a low-rank subspace of the full space the uncompressed projection could represent, so little is lost by restricting the projection to that subspace during training rather than bolting compression on afterward.

The RoPE problem, and decoupled RoPE

There is a complication that makes MLA harder than "just factor the matrix." Rotary Position Embeddings (RoPE) are the dominant way modern LLMs encode token position: they rotate pairs of dimensions within the query and key vectors by an angle proportional to position, so that the dot product between a query and a key ends up depending only on their relative distance (Su et al., 2021, RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv:2104.09864). RoPE has to act on the actual key vector, after decompression, for the geometry to work out correctly. That collides with an important inference-time optimization: at serving time, DeepSeek's implementation algebraically absorbs the up-projection matrix \(W^{UK}\) into the query projection that follows it, so the full-size key vector is never materialized at all. If RoPE must be applied to a fully decompressed key vector, that fusion is impossible, and the entire point of caching a small latent instead of a full key vector collapses.

The fix, decoupled RoPE, carves out a small additional slice of the query and key vectors, 64 dimensions per head in DeepSeek-V2, dedicated purely to carrying rotary position information. That slice is generated directly from the latent vector without ever fully decompressing into per-head space, position-encoded with RoPE as usual, and then concatenated onto the compressed content vector before the attention dot product. The content half stays compressible and matrix-absorbable; the position half stays compatible with RoPE. Notably, the RoPE slice for keys is shared across all heads, MQA-style, while each query head gets its own RoPE slice, which saves further redundant computation (Banatt, On MLA).

flowchart TD
    Hidn["Hidden state h_t"] --> DownKV["Down-project KV latent (W^DKV)"]
    Hidn --> DownQ["Down-project Q latent (W^DQ)"]
    DownKV --> LatKV["Latent vector, 512 dims"]
    DownQ --> LatQ["Latent vector, 1536 dims"]
    LatKV --> UpK["Up-project content K (W^UK)"]
    LatKV --> UpV["Up-project content V (W^UV)"]
    LatQ --> UpQ["Up-project content Q (W^UQ)"]
    Hidn --> RopeK["Shared RoPE key slice, 64 dims"]
    LatQ --> RopeQ["Per-head RoPE query slice, 64 dims"]
    UpK --> CatK["Concatenate content plus position"]
    RopeK --> CatK
    UpQ --> CatQ["Concatenate content plus position"]
    RopeQ --> CatQ
    CatK --> Score["Scaled dot-product attention"]
    CatQ --> Score
    UpV --> Score
    Score --> Out["Attention output"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff

    class Hidn blue
    class LatKV,LatQ amber
    class UpK,UpV,UpQ,RopeK,RopeQ purple
    class CatK,CatQ slate
    class Score,Out teal

[IMAGE: Annotated diagram of the decoupled RoPE split, showing a query/key vector divided into a shaded "content" segment and a hatched "positional" segment, with rotation arrows drawn only on the positional part]

What only gets cached

The practical payoff is simple to state: at inference time, only \(c_t^{KV}\) (512 dimensions) plus the shared RoPE key slice (64 dimensions) need to persist in the KV cache per token per layer, a total of 576 dimensions, versus 32,768 numbers for full MHA at DeepSeek-V2's head configuration. Everything else, the per-head expansions, gets recomputed from the cached latent at the moment attention runs.

Seeing It in Motion

sequenceDiagram
    participant Clnt as Client
    participant Route as Router
    participant Prefl as Prefill Engine
    participant Decod as Decode Engine
    participant Latnt as Latent Cache

    Clnt->>Route: Send prompt, 200 tokens
    Route->>Prefl: Forward for prefill pass
    Prefl->>Prefl: Compute compressed latent per layer
    Prefl->>Latnt: Store latent vectors, not full K or V
    Prefl->>Clnt: Return first generated token
    loop Each new token
        Clnt->>Decod: Request next token
        Decod->>Latnt: Read cached latent vectors
        Decod->>Decod: Up-project to per-head K and V on demand
        Decod->>Clnt: Return next token
    end
    Note over Latnt: Cache holds compressed vectors only, reconstruction happens at read time

Two engines, one shared cache, and the compression happening once during prefill rather than being repeated on every decode step is what keeps decode-time memory bandwidth low. This is also where MLA's real cost shows up: the up-projection in the decode loop is extra matrix multiplication that a cache holding raw, uncompressed keys and values would not need to pay.

flowchart TD
    Start["New model: choose an attention mechanism"] --> Q1{"Is decode-time memory bandwidth the bottleneck?"}
    Q1 -->|No| Full["Use standard multi-head attention"]
    Q1 -->|Yes| Q2{"Training from scratch, or adapting an existing checkpoint?"}
    Q2 -->|From scratch| Q3{"Is long-context quality critical?"}
    Q2 -->|Adapting a checkpoint| Trans["Use a TransMLA-style conversion"]
    Q3 -->|Yes| MLAOpt["Use Multi-Head Latent Attention"]
    Q3 -->|Quality bar is lower| GQAOpt["Use Grouped-Query Attention"]

    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class Start,Q1,Q2,Q3 slate
    class Full blue
    class Trans amber
    class MLAOpt,GQAOpt emerald

[IMAGE: Decision-tree poster version of the above flowchart, redrawn as a clean editorial infographic with icon-style nodes]

By the Numbers

The DeepSeek-V3 training-cost figure is explicitly a theoretical rental-price calculation covering only the final training run; it excludes prior research, ablations, and the hardware's actual capital cost, and DeepSeek's own paper and independent commentary both flag this (DeepSeek-AI, 2024, arXiv:2412.19437). Treat it as a lower-bound estimate of marginal compute cost, not a total cost of ownership.

[IMAGE: Bar chart comparing KV cache size per token in kilobytes across MHA, MQA, 8-way GQA, and MLA for a 128-head, 128-dimension model, log-scale y-axis, annotated with the 93.3% reduction DeepSeek reports]

[IMAGE: Line plot of maximum generation throughput in tokens per second for DeepSeek 67B versus DeepSeek-V2 across increasing batch sizes, annotated at the 5.76x peak]

A Concrete Example

Take DeepSeek-V2's published configuration directly and work through what gets stored for a single token at a single layer, in bytes, using 16-bit precision (2 bytes per number).

Standard multi-head attention, with \(n_h = 128\) heads and \(d_h = 128\) dimensions per head, stores a full key and a full value vector per head:

\[\text{MHA cache} = 2 \times n_h \times d_h \times 2\ \text{bytes} = 2 \times 128 \times 128 \times 2 = 65{,}536\ \text{bytes} \approx 64\ \text{KB}\]

MLA instead stores only the compressed KV latent vector, \(d_c = 512\) dimensions, plus the shared RoPE key slice, \(d_h^R = 64\) dimensions:

\[\text{MLA cache} = (d_c + d_h^R) \times 2\ \text{bytes} = (512 + 64) \times 2 = 1{,}152\ \text{bytes} \approx 1.125\ \text{KB}\]

That is a reduction of $1 - 1{,}152/65{,}536 \approx 98.24\%$ for this single layer, arithmetically, from the published hyperparameters alone. It lines up closely with the roughly 98.6% reduction that independent profiling of a similarly configured MLA model measured directly from running code, with the small gap plausibly explained by differences in exactly which auxiliary tensors each calculation counts (Banatt, On MLA). DeepSeek's own headline figure of 93.3% is measured end-to-end against their earlier dense 67B model rather than against an apples-to-apples MHA version of V2 itself, which is why it is a smaller percentage than the layer-level arithmetic above; it reflects a real product comparison rather than a controlled ablation.

Scale this out to a full request. For a 128K-token context, DeepSeek-V2's maximum supported length, and a 60-layer model, the MHA-style cache would need roughly $65{,}536 \times 128{,}000 \times 60 \approx 503$ GB per sequence, which is far beyond what a single accelerator or even a small cluster could hold for one user, let alone a batch of concurrent users. The MLA-style cache for the same sequence needs roughly $1{,}152 \times 128{,}000 \times 60 \approx 8.8$ GB, comfortably fitting alongside model weights on realistic serving hardware. That gap, not the exact percentage, is the entire economic argument for MLA at long context.

[IMAGE: Annotated code snippet rendered as a figure, showing the down-projection and up-projection matrix multiplications for MLA with matrix dimensions labeled at each arrow]

Where It Breaks

MLA's compression is not a free lunch, and the honest accounting matters more here than in the marketing summary.

Extra compute for saved memory. Every attention call now requires up-projecting the latent vector back into per-head keys and values, adding matrix multiplications that a model with a full, uncompressed cache does not need. Independent re-implementations of the reference DeepSeek-V2 architecture found that the open-sourced modeling code actually uses full KV caching rather than caching the compressed latent as the paper describes, because decompressing at every step is slower when memory is not the binding constraint. Compressed caching only wins once you are memory-bandwidth-bound, which is the common case in large-batch serving but not universal (Banatt, On MLA).

RoPE is genuinely awkward, and decoupled RoPE is a workaround, not a clean solution. The 50/50 split between content and position dimensions in DeepSeek-V2's configuration is a hyperparameter, and it is unclear whether that ratio is optimal at every model scale; a small-scale replication found the interaction between the decomposition and the RoPE split sensitive enough that results varied by hyperparameter choice in ways that are still not fully mapped (Banatt, On MLA).

Quality claims are strongest at large scale and murkier at small scale. Small controlled ablations (35M and 300M-parameter models) found MLA competitive with, but not clearly superior to, MHA, and in some RoPE configurations, standard MHA slightly outperformed MLA on training perplexity. DeepSeek's own reported superiority is measured through downstream benchmark scores on models with over 20B active parameters, not through validation perplexity on directly comparable small models, which leaves open whether the "improves on MHA" claim generalizes down to smaller scales or whether it is an emergent property of MLA's interaction with DeepSeek's Mixture-of-Experts architecture specifically (Banatt, On MLA).

It is not the automatic right choice. MiniMax explicitly tested hybrid and efficient-attention variants for its M2 model and reverted to full multi-head attention across all layers, reporting that no efficient variant they tried reliably matched full attention quality on retrieval, multi-hop reasoning, and agentic tool-use tasks, especially after supervised fine-tuning at long context (MiniMax, 2026, arXiv:2605.26494). That is a direct, recent counter-example to the assumption that KV cache compression is always worth the tradeoff, from a team that tried it and walked back.

Alternative Designs

TransMLA occupies an interesting middle ground: rather than choosing MLA from the start, it converts an already-trained GQA model into an MLA-shaped one after the fact, reporting a 93% KV cache reduction and 10.6x inference speedup on LLaMA-2-7B at 8K context after roughly 6 billion tokens of recovery fine-tuning (Meng et al., 2025, arXiv:2502.07864). That makes MLA-style compression available to teams that already have a GQA checkpoint and do not want to retrain from scratch, though it is a retrofit, not a design choice made from first principles.

How It Is Used in Practice

DeepSeek shipped MLA in production twice: first in DeepSeek-V2 (236B total, 21B active), then at far larger scale in DeepSeek-V3 (671B total, 37B active), where MLA and the DeepSeekMoE sparse-expert architecture together are credited with keeping training cost down to roughly $5.576M in compute at assumed 2024 H800 rental prices, alongside efficient inference at deployment (DeepSeek-AI, 2024, arXiv:2412.19437). The technique has since spread beyond DeepSeek's own lab. Moonshot AI's Kimi K2, a 1-trillion-parameter, 32B-active MoE model, adopts MLA directly, citing the same rationale: KV cache footprint matters disproportionately for long-context and multi-agent serving workloads, where many concurrent sessions each hold a large context in memory simultaneously (Kimi Team, 2025, arXiv:2507.20534). Zhipu's GLM-5 builds on DeepSeek Sparse Attention (DSA), a sparse token-selection layer that itself descends from MLA's compression approach, to control both training and inference cost while preserving long-context fidelity (GLM-5 Team, 2026, arXiv:2602.15763).

Engineering teams adopting MLA in serving stacks have had to build supporting infrastructure rather than treating it as a drop-in change: vLLM and SGLang both added MLA-specific kernels and cache layouts to realize the memory savings in practice, and hardware-level analysis has shown that MLA shifts attention workloads from memory-bandwidth-bound toward compute-bound, which changes which accelerators are best suited to running it and how batching should be tuned (Geens and Verhelst, 2025, arXiv:2506.02523). This is a meaningful operability point: a technique that changes the fundamental bottleneck of a workload also changes the autoscaling, batching, and hardware-selection logic built around the old bottleneck, and teams that treat MLA as a pure drop-in memory win without re-tuning their serving stack tend to leave performance on the table.

[IMAGE: Before/after stacked bar chart of GPU memory usage for a 128K-context request served with GQA versus MLA cache, separate stacked segments for model weights versus KV cache]

Insights Worth Remembering

• The KV cache, not the model weights, is frequently the real memory constraint at long context and high concurrency; architecture choices that shrink the cache can matter more for serving cost than choices that shrink the model.
• MLA is a genuinely different move from MQA and GQA: those methods reduce the number of distinct heads, while MLA keeps every head but compresses what gets stored between them, only reconstructing full size on demand.
• Rotary position embeddings and low-rank KV compression are fundamentally in tension, because RoPE needs to touch content that compression wants to hide; decoupled RoPE is a specific, non-obvious engineering answer to that tension, not a generic patch.
• A technique that reduces memory bandwidth demand while adding compute changes which hardware is the right fit; MLA can shift a workload from memory-bound to compute-bound, and serving infrastructure has to be re-tuned around that shift, not just pointed at a smaller cache.
• Benchmarks reported by the team that invented a technique deserve real weight, since DeepSeek's numbers are reproducible and widely cited, but they are strongest at the scale DeepSeek tested; independent small-scale replication found a murkier, more contested picture than the headline claim suggests.
• The field has not converged. Kimi K2 and GLM-5's lineage lean into MLA and MLA-derived sparse attention, while MiniMax explicitly measured the alternatives and chose full attention instead for its M2 model, a genuine, recent disagreement among frontier labs about where the real tradeoff lies.
• TransMLA demonstrates that architecture choices made at pretraining time are not necessarily permanent; a GQA model can be converted toward MLA-style compression after the fact with a comparatively small amount of continued training.

Open Questions

It is established, from DeepSeek's own reported figures and from independent hardware analysis, that MLA meaningfully reduces KV cache size and can shift attention workloads toward being compute-bound rather than memory-bound (DeepSeek-AI, 2024, arXiv:2405.04434; Geens and Verhelst, 2025, arXiv:2506.02523). It is also established that TransMLA can retrofit this compression onto an existing GQA checkpoint with a relatively modest fine-tuning budget (Meng et al., 2025, arXiv:2502.07864).

What remains genuinely open: whether MLA's claimed quality edge over MHA is a general property of the mechanism or an artifact of its interaction with DeepSeek's specific Mixture-of-Experts architecture and training recipe is not yet settled by independent replication at matched scale. Whether the 50/50 content-versus-position split in decoupled RoPE is close to optimal, or whether different ratios would recover more quality at a given cache size, is an unresolved hyperparameter question rather than a solved one. Whether MLA-style compression is compatible with the newer generation of extremely sparse attention mechanisms without compounding quality loss is an active area, and GLM-5's combination of MLA-descended compression with DeepSeek Sparse Attention is one live data point rather than a settled answer. Finally, MiniMax's decision to walk back efficient attention entirely for M2 suggests the field has not reached consensus on whether KV cache compression techniques as a category are worth their quality risk for reasoning-heavy and agentic workloads specifically, as distinct from general-purpose chat and retrieval workloads where the case for compression is stronger.

Sources and Further Reading

Foundational Papers

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762
• Su et al., 2021, RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv:2104.09864
• Shazeer, 2019, Fast Transformer Decoding: One Write-Head is All You Need, arXiv:1911.02150
• Ainslie et al., 2023, GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, arXiv:2305.13245
• DeepSeek-AI, 2024, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, arXiv:2405.04434

Important Follow-up Work

• DeepSeek-AI, 2024, DeepSeek-V3 Technical Report, arXiv:2412.19437
• Meng et al., 2025, TransMLA: Multi-Head Latent Attention Is All You Need, arXiv:2502.07864
• Geens and Verhelst, 2025, Hardware-Centric Analysis of DeepSeek's Multi-Head Latent Attention, arXiv:2506.02523
• Kimi Team (Moonshot AI), 2025, Kimi K2: Open Agentic Intelligence, arXiv:2507.20534
• GLM-5 Team, 2026, GLM-5: from Vibe Coding to Agentic Engineering, arXiv:2602.15763
• MiniMax, 2026, The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence, arXiv:2605.26494

Technical Blogs

• Banatt, Eryk, On MLA, independent ablation experiments and inference-time benchmarking of MLA versus MHA and MQA

A