Titans: The Sequence Architecture That Learns to Remember While It Runs

A Transformer never really remembers anything. It re-reads. Every token in the context window gets compared against every other token, at every layer, on every forward pass, which is why a 100,000-token prompt costs roughly ten thousand times more than a 1,000-token one under plain softmax attention. The alternative that recurrent models offer, folding the past into a fixed-size hidden state, is cheap but leaky: squeeze a novel into a vector of a few thousand numbers and something has to give. In January 2025, a team at Google Research proposed a third option that neither camp had really tried: give the model a memory that is itself a small neural network, and let that network keep learning, adjusting its own weights, while it processes the very sequence you handed it (Behrouz, Zhong, and Mirrokni, 2025, Titans: Learning to Memorize at Test Time, arXiv:2501.00663).

Why this matters: Every long-context system today makes the same bet, that a fixed-size state (a KV cache, a Mamba state, an RNN hidden vector) can hold enough of the past to answer questions about it later. Titans questions that bet directly by making the state a learner instead of a container, and reports needle-in-a-haystack accuracy that keeps climbing past two million tokens where fixed-size baselines flatten out.

TL;DR

• Titans replaces the fixed-size compressed state of RNNs and state-space models with a deep neural network (a multi-layer perceptron) that memorizes by taking a gradient step on every new token, not by architecture alone but as a first-class inference-time computation.
• The mechanism borrows a "surprise" signal from cognitive psychology: tokens that violate the memory's current predictions produce large gradients and get written strongly; expected tokens produce small gradients and are mostly skipped.
• Momentum carries "past surprise" forward so a token that follows a surprising one still gets remembered, even if it is individually mundane, and an adaptive decay term acts as a forgetting gate so the fixed-capacity memory does not overflow.
• Three ways to wire this memory into a model were tested: Memory as Context (MAC), Memory as Gate (MAG), and Memory as Layer (MAL). MAC and MAG both beat MAL, and MAC handles longer-range dependencies best.
• On the BABILong benchmark, which requires chaining facts scattered across extremely long documents, Titans outperformed much larger models including GPT-4, and the paper reports effective scaling past 2 million tokens on needle-in-a-haystack retrieval.
• The idea has already spawned a small research program: MIRAS generalizes the framework theoretically, ATLAS optimizes the memory update itself, and a follow-up called Nested Learning reframes deep networks as stacks of optimization problems running at different speeds.
• The closest sibling idea, TTT layers from a separate 2024 paper, already shipped inside a real generative system: a CVPR 2025 paper used test-time-training layers to generate coherent one-minute videos, beating Mamba-2 and Gated DeltaNet baselines by a wide margin in human evaluation.

At a Glance

flowchart LR
  Input["Input tokens"] --> Split["Segment into chunks"]
  Split --> Persist["Persistent memory, fixed task knowledge"]
  Split --> Neural["Neural long-term memory, updates during the forward pass"]
  Persist --> Core["Core: bounded attention window"]
  Neural --> Core
  Core --> Output["Output"]
  Core --> Neural
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  class Input blue
  class Split slate
  class Persist slate
  class Neural purple
  class Core teal
  class Output teal

Three memory systems run side by side here: a fixed set of learned parameters that never change after training (persistent memory), a small network that keeps adapting to whatever sequence it is currently reading (neural long-term memory), and a bounded attention window that handles precise short-range recall (the core). The arrow looping back from the core into the neural memory is the detail that makes Titans different from everything that came before it: the memory is still being written to as the model answers.

[IMAGE: Side-by-side schematic comparing a Transformer's growing KV cache, a Mamba state vector, and a Titans neural memory module, each annotated with what happens to it as sequence length increases]

Before Titans

When the Transformer arrived, its authors trained it on sentence pairs, and attention's quadratic cost was a minor footnote (Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762). It stopped being a footnote once people wanted models to read books, codebases, and legal contracts. Two rescue strategies emerged. One kept attention but made it cheaper: sparsify it, approximate the softmax, or replace it with a kernel function so the whole thing reduces to a recurrence over a matrix-valued state. The other abandoned attention for state-space models, most visibly Mamba, which processes a sequence in linear time by carrying a fixed-size state forward and updating it with a selective, input-dependent rule (Gu and Dao, 2023, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv:2312.00752).

Both strategies share an assumption: the past gets compressed into a fixed-size state, updated by a simple, usually additive, rule. That additive rule was already a known liability. Schlag, Irie, and Schmidhuber showed linear attention is formally equivalent to a "fast weight program," where a slow network writes instructions, additive outer products of keys and values, into a fast network's weights (Schlag, Irie, and Schmidhuber, 2021, Linear Transformers Are Secretly Fast Weight Programmers, arXiv:2102.11174). Purely additive writes never overwrite anything, so the state fills up and old associations get crowded out. The fix was the delta rule: before writing a new key-value pair, subtract out whatever the state currently associates with that key, so the write corrects rather than adds. Gated DeltaNet combined this delta rule with a learned forgetting gate and beat Mamba-2 on language modeling, reasoning, and long-context retrieval (Yang, Kautz, and Hatamizadeh, 2024, Gated Delta Networks: Improving Mamba2 with Delta Rule, arXiv:2412.06464).

Every step in this lineage improved how the state gets written to; none questioned what kind of object it should be. A vector or matrix, however cleverly updated, is a linear surface, able to represent only associations that are linearly separable in whatever space the keys live in. A separate line of work asked a more radical question: what if the hidden state were a small neural network instead, expressive enough to fit nonlinear patterns, updated by a literal gradient-descent step applied at test time, on the sequence being processed right now (Sun et al., 2024, Learning to (Learn at Test Time): RNNs with Expressive Hidden States, arXiv:2407.04620)? Titans took that idea and built a full architecture, with an attention core, around it.

timeline
  title From fixed-size recurrence to memory that learns at test time
  2017 : Transformers formalize attention as associative memory, quadratic cost
  2020 : Linear attention compresses context into a fixed matrix state
  2021 : Fast weight programmers reframe linear attention as a written program
  2023 : Mamba brings selective state spaces, linear-time sequence modeling
  2024 : Delta rule and gating improve what gets written to the state, TTT layers make the state a trainable model
  2025 : Titans and MIRAS make the memory a deep network that learns online, ATLAS and Nested Learning follow

How Titans Actually Works

The memory as an online learner, not a container

Reframe what a recurrent model's hidden state is for. In the standard view, a state \(\mathcal{M}\) is updated by a write function and queried by a read function: \(\mathcal{M}_t = f(\mathcal{M}_{t-1}, x_t)\), then \(y_t = g(\mathcal{M}_t, x_t)\). Titans keeps that two-step shape but changes what \(\mathcal{M}\) is. Instead of a vector or matrix, \(\mathcal{M}\) is a multi-layer perceptron, parameterized by its own weights, and instead of a hand-designed write rule, the write step is a gradient update on an explicit loss. Each incoming token is projected into a key and a value, \(k_t = x_t W_K\) and \(v_t = x_t W_V\), exactly as in attention. The memory is trained, online, to associate keys with values by minimizing an associative memory loss:

\[\ell(\mathcal{M}_{t-1}; x_t) = \left\lVert \mathcal{M}_{t-1}(k_t) - v_t \right\rVert_2^2\]

This is the same objective an ordinary neural network minimizes during training, an L2 regression loss between a prediction and a target, except the "training set" is the sequence itself, one token at a time, and the "training" happens during inference. Reading from memory later is just a forward pass with no weight update: given a query \(q_t = x_t W_Q\), the output is \(y_t = \mathcal{M}_t(q_t)\).

The surprise metric, momentum, and forgetting

The gradient of that loss with respect to the memory's current weights is what the paper calls the momentary surprise: how wrong the memory currently is about this particular token. A token the memory already predicts well produces a small gradient and barely nudges the weights; a token that violates the memory's current associations produces a large gradient and forces a substantial update. This is a direct computational analogue of a well-documented property of human memory: routine events fade, but events that break an expectation are what stick (Behrouz, Zhong, and Mirrokni, 2025).

Momentary surprise alone has a blind spot. Consider a paragraph that opens with one genuinely surprising sentence followed by several sentences that are, in isolation, unremarkable but causally connected to the surprising one. A memory that only reacts to the gradient of the current token would write the first sentence and then let the connective tissue slip by. Titans addresses this with a momentum term, so what gets written depends on both the current gradient and an exponentially weighted trace of recent gradients:

\[S_t = \eta_t S_{t-1} - \theta_t \, \nabla \ell(\mathcal{M}_{t-1}; x_t)\]

Here \(\eta_t\) is a data-dependent momentum coefficient and \(\theta_t\) a data-dependent learning rate; both are produced by small learned projections of the input, so the model controls its own plasticity token by token. Finally, because the memory has finite capacity, the paper adds an adaptive decay term \(\alpha_t \in [0, 1]\) that shrinks old weights before the new surprise term is folded in:

\[\mathcal{M}_t = (1 - \alpha_t)\, \mathcal{M}_{t-1} + S_t\]

The authors show this decay is a generalization of the forget gates already used in Mamba-2 and similar linear recurrent models; the difference is that here it is gating the weights of a deep network rather than the entries of a matrix. Read as a whole, the update rule is mini-batch gradient descent with momentum and weight decay, run continuously over the input stream, which is precisely the training loop of an ordinary neural network, just compressed into a per-token forward pass.

[IMAGE: Annotated three-token trace showing key/value projection, loss value, and gradient magnitude for a routine token, a surprising token, and a post-surprise token, with the momentum term visibly carried forward across the three columns]

flowchart TD
  Token["New token"] --> KeyVal["Project to key and value"]
  KeyVal --> Loss["Associative loss: memory recall vs actual value"]
  Loss --> Grad["Momentary surprise: gradient of the loss"]
  Grad --> Mom["Combine with past surprise via momentum"]
  Mom --> Decay["Apply adaptive forgetting, weight decay"]
  Decay --> Update["Write new memory weights"]
  Update --> Recall["Later query reads memory, no further update"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  class Token blue
  class KeyVal blue
  class Loss amber
  class Grad amber
  class Mom purple
  class Decay purple
  class Update teal
  class Recall teal

Making test-time gradient steps fast enough to matter

Running literal gradient descent, one token at a time, on hardware built for large matrix multiplications sounds like a performance disaster, and the naive version would be. The paper's practical contribution tensorizes this recurrence into chunked matrix operations rather than a token-by-token loop. The sequence is split into segments; within a segment, the data-dependent coefficients (\(\eta_t\), \(\theta_t\), \(\alpha_t\)) are simplified to be constant, turning the update into a linear time-invariant system computable with the same associative-scan machinery that makes Mamba-2 fast. The tradeoff is explicit: constant-within-chunk parameters sacrifice token-level expressiveness for parallel, hardware-friendly training, the same bargain every hardware-aware recurrent architecture makes, and why Titans reports training throughput comparable to modern linear recurrent baselines rather than something dramatically slower.

Persistent memory, and where attention fits

A neural memory that only learns from the current input has no room for fixed, task-level knowledge, the kind of thing a model should know regardless of what document it is reading right now. Titans adds a small set of learnable, input-independent vectors, persistent memory, prepended to every sequence like a permanent set of context tokens. The authors describe three roles for these tokens: storing task abstractions the input-dependent memory cannot hold, functioning analogously to a Transformer block's feedforward layers, and empirically reducing attention's tendency to over-allocate weight to a sequence's first few tokens simply because they are first.

Attention itself does not disappear; it becomes the model's short-term, high-precision memory, handling exact recall within a bounded window while the neural memory handles everything older. The paper tests three ways of wiring these three components (persistent memory, neural long-term memory, and a bounded attention core) together, and which wiring you choose changes what the model can and cannot do well.

Memory as a Context (MAC)

The input is split into segments. For each segment, the model first queries the neural memory for a summary of relevant history, concatenates that summary with the persistent memory tokens and the segment itself, and only then runs full attention over the combined sequence. Attention, seeing the retrieved summary sitting right next to the current tokens, effectively decides whether the historical context is worth attending to at all. The neural memory continues updating after this attention pass, so the loop is genuinely closed: history informs the present, and the present, having been processed, updates history.

Memory as a Gate (MAG)

Here the input is not segmented. Two branches process it in parallel: a sliding-window attention branch handles short-term, precise recall, and the neural memory branch updates continuously on the raw stream. A learned nonlinear gate then combines the two outputs. This is structurally closer to a mixture of two experts, one precise and local, one persistent and global, than to MAC's sequential retrieve-then-attend pipeline.

Memory as a Layer (MAL)

The simplest wiring: the neural memory is just another layer, compressing the input before a sliding-window attention layer processes what comes out. It is easy to implement and reason about, but the paper's own results show it is the weakest of the three, because compressing first and attending second means attention never gets to see the raw signal, only whatever the memory chose to keep.

sequenceDiagram
  participant Segm as Input Segment
  participant Persm as Persistent Memory
  participant Ltm as Long-Term Memory
  participant Attn as Core Attention
  participant Outp as Output Stream

  Segm->>Ltm: Query long-term memory for relevant history
  Ltm-->>Segm: Return retrieved summary vector
  Segm->>Attn: Concatenate segment, persistent tokens, and summary
  Persm->>Attn: Prepend fixed task-knowledge tokens
  Attn->>Outp: Emit refined representation
  Attn->>Ltm: Pass tokens forward for the next surprise calculation
  Ltm->>Ltm: Update memory weights with a gradient step
  Note over Ltm: The update happens inside the forward pass, no offline retraining

Seeing It in Motion

The sequence diagram above traces one segment through the MAC variant. The diagram below lays the three wirings side by side so the structural difference is visible at a glance: MAC's sequential retrieve-then-attend pipeline, MAG's parallel-then-gate structure, and MAL's compress-then-attend stack.

graph TD
  subgraph MacBox["MAC: Memory as Context"]
    M1["Long-term memory retrieves history"] --> M2["Concatenated with persistent tokens"] --> M3["Full attention over combined context"]
  end
  subgraph MagBox["MAG: Memory as Gate"]
    G1["Sliding-window attention branch"] --> G3["Nonlinear gate combines branches"]
    G2["Long-term memory branch"] --> G3
  end
  subgraph MalBox["MAL: Memory as Layer"]
    L1["Long-term memory compresses input"] --> L2["Sliding-window attention on compressed sequence"]
  end
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  class M1 purple
  class M2 slate
  class M3 teal
  class G1 teal
  class G2 purple
  class G3 slate
  class L1 purple
  class L2 teal

[IMAGE: Annotated bar chart of accuracy on the BABILong benchmark across sequence lengths from 1K to over 1M tokens, comparing Titans against GPT-4, Mamba-2, and a retrieval-augmented baseline, with the Titans line staying flat where others degrade]

[IMAGE: Heatmap of memory-write magnitude (the surprise gradient norm) over a token sequence, with a visibly bright band at a plot twist in a story and dim bands over routine dialogue, to visualize what "surprising" looks like in practice]

By the Numbers

The headline experimental claims, as reported in the paper and corroborated by Google Research's own summary, are qualitative more often than they are exact digits the authors published as a single clean table; where a precise figure exists it is cited, and everything else is described the way the source describes it.

Two claims deserve more caution than the rest. "Beyond 2 million tokens" and "outperforms GPT-4" both come from the primary source and its own summary, but neither publishes exact per-length accuracy figures in an easily tabulated form, so this article reports the comparative claim rather than inventing precision the sources do not offer. The ablations do report one unambiguous, mechanistic finding: at matched parameter counts, deeper memory MLPs achieve lower perplexity than shallower ones, and the gap widens with sequence length, direct evidence that depth in the memory module is doing real work, not just adding capacity for its own sake.

A Concrete Example

Picture a Titans (MAC) model reading a 50,000-token merger agreement, split into 2,048-token segments, roughly 24 segments end to end. Segment 14 is routine boilerplate about notice periods and governing law, language that appears in nearly every contract the memory has trained on. Segment 15 opens with an unusual indemnification clause: the seller retains liability for a category of environmental claims for eighteen years after closing, far outside the market-standard three-to-five-year survival period.

Walk segment 15 through the mechanism (the gradient magnitudes below are an illustrative walkthrough for intuition, not numbers published in the paper). For the routine opening tokens, the memory's current weights already predict the associated values well, so the loss \(\ell(\mathcal{M}_{t-1}; x_t)\) and its gradient are small, and the update to \(\mathcal{M}\) is barely perceptible. When the model reaches "eighteen years," a term the memory has essentially never associated with "indemnification survival period," the loss spikes, the gradient is large, and the momentum term \(S_t\) receives a strong push. Momentum matters because the next dozen tokens, connective legal language like "notwithstanding Section 9.2," are individually mundane; momentary surprise alone would let them slip by nearly unwritten, but \(S_t\) still carries the push from the surprising phrase, so the memory keeps writing with elevated strength through the whole clause.

Five segments and roughly 10,000 tokens later, segment 20 references "the environmental indemnification obligations described above." Now retrieval, not writing, matters: the query projection \(q_t\) is close, in the memory's learned key space, to the key written during segment 15's clause. The read \(y_t = \mathcal{M}_t(q_t)\) pulls out a representation shaped by that earlier update, with no need for segment 15's raw tokens to still sit in an attention window (a sliding-window model with a window under 10,000 tokens would have no access to segment 15 at all by now). Meanwhile the decay term has quietly shrunk the memory's weights across all 20 segments, so routine boilerplate has faded proportionally more than the indemnification clause, whose large writes partially offset the decay. This is the concrete payoff of a fixed-capacity, continuously learning memory over a fixed-size vector: what gets preserved is chosen by how surprising it was, not by how recently it arrived.

[IMAGE: Horizontal timeline schematic of the 24-segment merger agreement, with segment 15 marked as a bright memory-write spike, segment 20 marked with a dashed retrieval arrow pointing back to segment 15, and the intervening segments shown fading in a lighter shade to represent decay]

Where It Breaks

The mechanism is not free of real costs and real limitations, and the paper and its critics are reasonably direct about several of them.

Every token now triggers a small training step: projection into keys and values, a loss computation, a gradient, a momentum update, and a decayed write, rather than a single matrix multiply. Chunked, tensorized implementation makes this practical on accelerators, but it is not free; it costs more compute per token than a Mamba-2-style linear update, and the paper's own efficiency comparisons position Titans as competitive with, not categorically cheaper than, other linear recurrent architectures.

The chunking trick that makes training fast requires treating the learning rate, momentum coefficient, and decay rate as constant within a chunk, a real expressiveness tax. The fully data-dependent, token-by-token version of the update rule is more powerful in principle than the version actually trained at scale, and the gap between the two is not extensively quantified in the paper.

There is also an inherent tension between memorization and generalization that "online meta-learning" manages rather than dissolves. A memory that adapts too eagerly risks overfitting to the exact sequence at hand and failing to generalize to a slightly different phrasing of the same fact; one that adapts too conservatively is just a slow linear recurrence with extra steps. The learned surprise thresholds are the model's attempt to sit in the right place on that spectrum, but "the right place" is task-dependent and the benchmarks do not stress-test this directly.

The surprise mechanism is also a proxy for salience, not a guarantee of it. A gradient-based novelty signal reliably fires on statistically unusual tokens, but human-relevant importance and statistical unlikelihood are not the same thing. A subtly misleading but statistically ordinary sentence, exactly the kind that matters most in adversarial settings, could produce a small gradient and get written weakly even though a human reader would flag it immediately.

Finally, the empirical validation, broad across task types (language modeling, reasoning, genomics, time series, needle-in-a-haystack, BABILong), is still research scale: the main tables run up to roughly 760M parameters, and "test-time learning" here means learning within a single forward pass, not memory that persists automatically across separate sessions; carrying state across sessions would need the same checkpointing any stateful serving system requires.

Alternative Designs

Represent these fairly rather than as a strict ranking: full attention is not obsolete, it is simply the most expensive point on the curve, and for contexts that already fit in a practical cache budget, its exactness is hard to beat. State-space models remain the default choice when raw throughput matters more than long-range factual recall. Titans and its TTT relatives are the newest entrants, better positioned for tasks that specifically punish fixed-size compression, and correspondingly less battle-tested in production.

[IMAGE: Four small labelled schematics in a row, one per memory-update rule, additive write, delta-rule write, gated delta-rule write, and gradient-step write, each showing the same incoming key-value pair and how it changes the memory contents differently]

How It Is Used in Practice

Titans itself, as described in the January 2025 paper, is a research architecture; the paper reports NeurIPS 2025 acceptance and research-scale results, and there is no publicly documented case of a shipped, widely used frontier language model built on Titans layers as of this writing. What has moved faster into a working, published system is the sibling idea, test-time-training layers with an expressive hidden state. A CVPR 2025 paper used TTT layers inside a pretrained video diffusion Transformer to generate coherent one-minute videos from text storyboards, built on a Tom and Jerry cartoon dataset as a proof of concept, and reported the TTT-augmented model beating Mamba-2, Gated DeltaNet, and sliding-window attention baselines by 34 Elo points in a 100-video human evaluation (Dalal et al., 2025, One-Minute Video Generation with Test-Time Training, arXiv:2504.05298). The author list includes NVIDIA researchers alongside the original TTT paper's academics, a real, if narrow, signal that the idea has cleared the bar of interesting to a systems team with production incentives.

From the paper: TTT-augmented generation led sliding-window attention, Mamba-2, and Gated DeltaNet baselines by 34 Elo points in a 100-video human evaluation, the clearest published sign yet that an expressive, learning-at-test-time memory changes outcomes on a task real users can judge by eye.

[IMAGE: Before/after video-frame grid comparing four consecutive keyframes from a one-minute Tom and Jerry-style generation, sliding-window attention baseline on top losing character consistency, TTT-layer version on bottom keeping the character and scene coherent]

Google Research's own December 2025 summary of Titans, paired with the follow-up MIRAS framework, frames the pair as a theoretical and architectural foundation rather than a shipped product, explicitly positioning the contribution as advancing "the concept of test-time memorization" as a research direction (Google Research, 2025, Titans + MIRAS: Helping AI have long-term memory). The clearer signal is the pace of follow-up from the same group in under a year: a generalized theoretical framework (MIRAS), an improved memory-optimization method (ATLAS), and a reframing of deep learning as nested optimization problems (Nested Learning), each pushing toward architectures that keep "learning" after training ends, not just during it.

Insights Worth Remembering

A fixed-size state is not a law of recurrent computation, it is a design choice, and Titans shows the choice has room to move: the state can be a trained model rather than a container, provided you pay the training-time cost of running it during inference.

The "surprise" framing quietly resolves a tension that has dogged compressed-memory architectures since fast weight programmers: instead of asking how to compress everything equally, it asks what a perfect predictor would fail to predict, and writes memory in proportion to that failure, a genuinely different objective than uniform compression.

Momentum in a memory-write rule does something conceptually different from momentum in a training loop, even though the math is nearly identical: it is not smoothing noisy gradients toward a minimum, it is deciding which tokens near a surprising event also deserve to be remembered.

Persistent memory and neural long-term memory answer two questions that are easy to conflate: what the model knows regardless of input, versus what this particular input has taught it so far. Keeping them as separate components, rather than one blended state, is what lets each specialize.

The chunk-parallelization tradeoff, sacrificing per-token data-dependence for training speed, is the same bargain every fast recurrent architecture has struck since Mamba-2; Titans does not escape it, it just makes the thing being chunked more expressive.

A result like "beats GPT-4 on BABILong with far fewer parameters" says more about parameter efficiency on a specific structured-reasoning task than about general capability; reading it as a broad capability claim would overstate a narrower, real result.

The fastest path from research idea to working system was not the flagship architecture itself but a structural cousin, applied to a different modality by a team with a concrete deliverable; ideas here are traveling sideways into adjacent problems faster than into production language models.

Open Questions

What has been shown: deeper memory modules reduce perplexity at matched parameter counts in the paper's own ablations, and the advantage grows with sequence length, measured evidence that depth genuinely helps rather than a choice justified only by intuition (Behrouz, Zhong, and Mirrokni, 2025).

What remains an open engineering problem: whether the chunk-parallel training scheme scales to trillion-token pretraining at frontier parameter counts without the constant-within-chunk simplification eroding the gains seen at sub-billion-parameter scale. The paper does not test this regime, and nothing in its design guarantees the tradeoff stays favorable as sequence length and model size grow together.

What is genuinely speculative: whether "surprise" as a gradient-norm proxy is the right general-purpose notion of importance, or whether it needs task-specific correction, an explicit mechanism for catching subtly misleading but statistically ordinary information, before it can be trusted where missing the truly important token has real consequences.

[IMAGE: 2x2 labelled grid schematic of the MIRAS framework's four design axes, memory architecture, attentional bias, retention gate, memory algorithm, with Titans, Transformers, and Mamba-2 each plotted as a small icon showing which choice they make on each axis]

An actively developing thread: Nested Learning suggests that how many timescales of optimization a model should run, and how they should be nested, is a design space as large as architecture search once was for layer types, implying the current three-tier split (persistent, neural long-term, attention) may be one point in a larger space rather than an endpoint (Behrouz, Razaviyayn, Zhong, and Mirrokni, 2025, Nested Learning: The Illusion of Deep Learning Architectures, arXiv:2512.24695). Whether any of this reaches frontier-scale deployed language models, rather than research-scale demonstrations and adjacent-modality systems like the video work, remains an open empirical question.

Sources and Further Reading

Foundational Papers

• Vaswani et al., 2017, Attention Is All You Need, arXiv:1706.03762
• Behrouz, Zhong, and Mirrokni, 2025, Titans: Learning to Memorize at Test Time, arXiv:2501.00663
• Schlag, Irie, and Schmidhuber, 2021, Linear Transformers Are Secretly Fast Weight Programmers, arXiv:2102.11174
• Sun et al., 2024, Learning to (Learn at Test Time): RNNs with Expressive Hidden States, arXiv:2407.04620

Important Follow-up Work

• Behrouz et al., 2025, It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization, arXiv:2504.13173
• Behrouz et al., 2025, ATLAS: Learning to Optimally Memorize the Context at Test Time, arXiv:2505.23735
• Behrouz, Razaviyayn, Zhong, and Mirrokni, 2025, Nested Learning: The Illusion of Deep Learning Architectures, arXiv:2512.24695
• Dalal et al., 2025, One-Minute Video Generation with Test-Time Training, arXiv:2504.05298
• Gu and Dao, 2023, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv:2312.00752
• Yang, Kautz, and Hatamizadeh, 2024, Gated Delta Networks: Improving Mamba2 with Delta Rule, arXiv:2412.06464
• Kuratov et al., 2024, BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack, arXiv:2406.10149

Technical Blogs

• Google Research, December 2025, Titans + MIRAS: Helping AI have long-term memory
• Andrey Lukyanenko, February 2025, Paper Review: Titans: Learning to Memorize at Test Time

Additional Resources

• BABILong benchmark repository
• Hugging Face paper page for Titans