ARTIFICIAL INTELLIGENCE
Titans and learning what to remember
Titans and the 2025 MIRAS framework turn memory into an online learning problem, raising precise questions about surprise, retention, and long-context evidence.
Research checked
Newest cited publication
A historical LSTM cell illustrates explicitly gated neural memory. It provides background for the distinct memory-learning mechanism proposed in Titans.
Guillaume Chevalier · Image source · CC BY-SA 4.0
A long conversation contains more information than a useful summary. Some details must be preserved exactly, others can be compressed, and many will never matter again. Titans approaches that problem by making part of a sequence model's memory a learner in its own right. Its original preprint arrived at the turn of 2025, and the subsequent MIRAS work offered a broader framework for understanding the design choices. The most interesting claim is not that a model now remembers like a person. It is that the rule for updating memory can become an explicit component that researchers can inspect, compare, and train. That shift creates opportunities for longer contexts while exposing a difficult question: who decides which experiences deserve the model's finite capacity?
Three memories with different responsibilities
The Titans preprint was submitted on 31 December 2024, despite its 2025 arXiv identifier. It combines an attention-based core, an adaptive long-term neural memory, and persistent parameters learned during training. The paper presents variants that incorporate memory as context, as a layer, or through a gated branch. Google Research's December 2025 explanation revisited the work alongside MIRAS. These dates identify an active research lineage; they do not establish a separately verified 2026 Titans deployment.
An independent analogy is a field researcher with a working desk, a changing notebook, and a familiar set of laboratory techniques. The desk holds the immediate material. The notebook condenses prior observations. Established techniques shape how both are used. The analogy helps distinguish roles, but biological memory and a neural module are not interchangeable. In particular, a compressed representation may influence an answer without preserving an inspectable record of the original observation. That matters when the task requires a quotation, provenance, or an exact number.
Surprise has a mathematical meaning
Titans uses the gradient of an associative-memory objective as a surprise signal. Its update combines current error, a momentum-like contribution from earlier updates, and adaptive forgetting. In this setting surprise means difficulty fitting an internal association, rather than emotional importance or factual novelty. The authors report experiments including language modeling and long-context retrieval, with needle-in-a-haystack tests extending beyond two million tokens. Those controlled retrieval results do not demonstrate lossless recall of every detail in arbitrary documents.
Consider an original stream containing routine temperature readings followed by a broken sensor's impossible value. The anomaly may produce a strong error signal, yet the desired action might be to reject it rather than remember it more strongly. Conversely, a quiet correction to a unit label might be crucial even if it looks familiar. This example explains why surprise and usefulness can diverge. A memory system needs an objective that reflects the eventual task, and any claim about selective retention should be tested on both meaningful exceptions and irrelevant disturbances.
An event is longer than one token
The official Google explanation emphasizes momentum as a way to connect current and recent surprise, and forgetting as a way to manage limited capacity. This helps explain why an update rule should not respond only to a single isolated input. An event may begin with an unexpected statement and continue with ordinary-looking details that explain it. The mechanism is still a learned numerical process; an account using human memory language should not imply that the module understands importance in a human sense.
For a toy example, suppose an equipment log announces that a pump has been replaced. The next lines give a serial number and a calibration offset. Those lines may be linguistically unsurprising but essential to interpreting later measurements. A rule that retains only the dramatic opening can miss the useful part. An ideal evaluation would place decisive details at different distances after an event and check whether the memory preserves the relationship. It would also insert distractions, preventing the model from succeeding merely by storing everything adjacent to an unusual token.
MIRAS makes the knobs explicit
MIRAS, submitted on 17 April 2025, frames sequence modeling through four choices: the memory structure, an internal objective called attentional bias, retention regularization, and the update algorithm. It introduces Moneta, Yaad, and Memora as different instantiations. The work connects architectural choices to online optimization and reports different strengths across evaluated tasks. MIRAS is a design framework, not simply another name for Titans, and its experiments do not license combining the best scores of separate variants into one claimed model.
This decomposition is valuable even without choosing a winning variant. Imagine two systems with the same memory size. One favors average reconstruction accuracy; another penalizes large isolated errors more strongly. Their summaries may look similar on ordinary data but behave differently on rare critical facts. A useful comparison would vary that internal objective while keeping the surrounding model fixed. Calling both systems recurrent would conceal the distinction. Naming the objective and the retention rule makes the experiment explainable and helps another team reproduce the relevant failure.
Forgetting is part of the specification
A finite memory cannot preserve unlimited independent information exactly. Compression works by taking advantage of regularities or by sacrificing detail. A longer demonstrated input length therefore needs to be paired with a statement about what remains recoverable. Retrieval of a distinctive inserted fact, multi-step reasoning across several facts, and faithful reconstruction of a whole document ask different things. This is an information-accounting argument, not a claim that the Titans experiments are invalid. Their interpretation becomes stronger when the required operation is stated precisely.
Forgetting can also be desirable. In a fictional monitoring system, a machine's old calibration should stop governing predictions after a verified replacement. Keeping both values indefinitely without knowing which applies could make performance worse. A responsible test would distinguish obsolete information from enduring background knowledge, and it would reveal whether the model's adaptation actually follows the change. Simply averaging accuracy across old and new tasks may hide the fact that some old answers are supposed to become wrong. The data stream needs timestamps and a clear target policy.
Context memory is not automatic personal memory
Online updates introduce a practical boundary question: which state belongs to one document, session, or user? A memory module that changes while reading a sequence does not automatically provide safe, durable personalization across sessions. An application must decide when to initialize, retain, export, or discard the state. That is an engineering inference from the existence of mutable memory, not a feature established by the research results. The policy becomes particularly important when evaluation assumes clean resets but a deployed service reuses a long-lived process.
An original audit could interleave two fictional customers whose documents contain conflicting project codes. After each switch, the evaluator would ask questions that reveal accidental state transfer. It would also check whether deleting one customer's saved record removes the corresponding influence, rather than merely hiding a visible note. Such tests would not establish every privacy property, but they would turn a vague promise about remembering into a concrete system behavior. Exact record storage and adaptive neural memory may need different controls because their contents are represented differently.
A memory result should name what survives
The verified papers show why learned memory deserves serious investigation. They make retention mechanisms trainable and compare concrete alternatives rather than treating the context window as the only available resource. The remaining uncertainty is substantial: compression can be task-dependent, surprise can favor distractions, and long input length can conceal weak recall of ordinary details. None of those concerns erases the reported results. They define the next experiments that would make the results easier to use.
I would prioritize a public evaluation with recurring corrections, distractors, exact quotations, and questions requiring several separated facts. It should track accuracy over time, memory size, update cost, and failures after many changes. Separate scores would show whether a model remembers an exception, understands when it applies, and abandons it when it becomes obsolete. That would connect architectural ingenuity to a clear account of useful memory: preserving the right relationship for the right period, with an honest description of what the system can no longer recover.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- Titans: Learning to Memorize at Test TimearXivPublished Checked
- It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online OptimizationarXivPublished Checked
- Titans + MIRAS: Helping AI have long-term memoryGoogle ResearchPublished Checked
FOLLOW THE QUESTION
Study engineering foundations