Muir Linear Attention Explained: 10-Million-Token Streams in a 25MB State

Muir linear attention explained — the Transformer architecture Muir builds on

Illustrative image: the Transformer architecture (Vaswani et al., 2017) that Muir's linear attention replaces. Image: Wikimedia Commons

TL;DR — Muir linear attention explained: Muir is a new AI architecture from Google Research and the University of Texas at Austin that reads 10-million-token streams while holding only a 25MB memory state. Posted on arXiv on 5 November 2026, it trains 2.7 times faster than a Transformer at 32K context and beats both Transformer and Gated DeltaNet-2 baselines — but it is a research preprint, with no public weights yet.

📌 KEY HIGHLIGHTS

Muir is a linear-attention model. Instead of storing every token it has seen, it compresses the past into a fixed-size state — so memory stays constant no matter how long the input gets.
The headline number: 10 million tokens. Trained on 4,096-token sequences, the 1.34-billion-parameter model reads streams 2,560 times longer without its predictions drifting — in a 25MB state where a same-size Transformer would need roughly a 2.1TB cache.
It also beats the baselines on quality. Perplexity is 1.8–2.0% lower than Gated DeltaNet-2 and 2.5–3.6% lower than a Transformer, with commonsense accuracy up by as much as 1.9 points.
Two ideas do the work: sensitivity-allocated low-rank gating and a matched-filter readout gate that stops loud, slow-decaying memories from drowning out quiet, distant ones.
Reality check: this is a preprint, not a peer-reviewed result. The weights have not been released, and the largest model tested is 1.34 billion parameters — research scale, not frontier scale.

What happened?

On 5 November 2026, researchers Tushaar Gangavarapu (University of Texas at Austin), Gil Tabak and Clemens JS Schaefer (Google Research) posted a paper on arXiv titled "Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout". The same day it reached number 14 on Hugging Face's Daily Papers, and independent recaps quickly followed.

Muir belongs to the linear-attention family — a line of research trying to fix the Transformer architecture's most expensive habit. A standard Transformer remembers everything it reads: every token's key and value are stored in a growing "KV cache", so the memory bill rises with every word. For short chats that hardly matters. For a model asked to read a whole codebase, a legal archive or a day of sensor logs, the cache becomes the bottleneck — the authors calculate that a same-size Transformer would need a 2.1TB cache to attempt the 10-million-token stream Muir handles in 25MB, and their own Transformer baseline ran out of chip memory at 128,000 tokens.

Linear attention removes that bottleneck by folding the past into a fixed-size matrix. Training time grows linearly instead of quadratically, and inference memory stays constant. The catch, historically, is recall: squeezing everything into a fixed state blurs exact details, so linear models have struggled to find a needle in a very long haystack. Muir is an attempt to fix exactly that weakness.

Muir linear attention explained: the two ideas inside it

The paper starts from a measurement, not a hunch. In gated delta-rule models — the family Muir builds on — the memory state is updated in two ways: an "erase" step removes the old prediction for an incoming key, and a "write" step stores the new value. The authors show the state is far less sensitive to erasure than to writes: key activations concentrate energy onto a sparse subset of channels, while value writes scale all channels. So the erase gate can be built at lower rank without losing accuracy.

That asymmetry drives Muir's first idea: sensitivity-allocated low-rank gating. All the gates are fused into a single low-rank projection, which cuts gate-projection cost by 4.5 times — and the saved parameters are reinvested in a wider feed-forward network, so the model spends its budget where it matters most.

The second idea tackles the other failure mode. Slow-decaying rows of the memory state accumulate much larger norms than fast rows, so when the model reads its memory back, the loud rows dominate and drown out quieter traces from earlier in the stream. The usual remedy is stronger erasure at write time — which discards the quiet traces entirely. Muir instead adds a matched-filter readout gate: it re-weights the state rows before reading, dialling down the channels that have grown loud and dialling up the ones that still carry distant signal. A new parallel update algorithm keeps training numerically stable at the large chunk sizes needed to saturate modern matrix hardware.

Muir linear attention explained — Muir Woods redwoods, the paper's namesake

Illustrative image: Muir Woods, California — the paper takes its name from the redwoods, whose rings hold centuries of history in a fixed trunk. Image: Wikimedia Commons / dalecruse, CC BY 2.0

The numbers that matter

All figures below are the authors' own measurements, from models between 166 million and 1.34 billion parameters:

Measure	Muir	Baseline
Memory at 10M tokens	25MB fixed state (float32)	~2.1TB KV cache for a same-size Transformer
Length extrapolation	Trained at 4K, tested to 10M tokens (2,560x) with no perplexity drift	Transformer baseline ran out of memory at 128K tokens
Passkey retrieval at 1M tokens	Near-100% with static readout	Gated DeltaNet-2 fails the same test at 4K–8K
Perplexity (language modelling)	1.8–2.0% lower than GDN-2; 2.5–3.6% lower than Transformer	GDN-2 and Transformer baselines
Commonsense accuracy	Up to +1.9 points over baselines	GDN-2 and Transformer baselines
Training speed	2.7x faster than Transformer at 32K context; reaches 100B-token validation loss 1.2x faster	Matches GDN-2 throughput

The long-context numbers are the showiest, but the quality numbers matter more for whether this design survives. A model that remembers everything but predicts worse is a curiosity; Muir claims to do both at once — better perplexity, better recall, and a fraction of the memory. The caveat is scale: 1.34 billion parameters is small by 2026 standards, and nothing here proves the gains hold at the tens or hundreds of billions of parameters where frontier models live.

Why it matters for readers in India

Long context is where AI meets India's paperwork. Legal research, medical records, government files, long codebases and multi-hour lecture transcripts are all "long-context" problems — and today they are expensive problems, because the KV cache makes serving them memory-hungry and costly. An architecture that holds a whole document in 25MB instead of terabytes would, if it scales, cut the serving cost of exactly the applications Indian startups and developers are building: document chatbots, code assistants that read entire repositories, and tutoring tools that remember a student's whole semester.

There is also a training-cost angle. Muir trains 2.7 times faster than a Transformer at 32K context on the same hardware. For Indian labs and startups that rent rather than own compute, faster training is directly cheaper training. But none of this is usable yet: with no released weights or code, Muir is a paper to watch, not a tool to download. The honest timeline is months — peer review, a public release, and independent replication — before anyone can build on it.

For students and early-career engineers, the paper is still worth reading now. Linear attention is becoming a core skill: our earlier explainer on the triangular Log-Det PDHG algorithm showed how a single algorithmic idea can change what is computable, and Muir is the same kind of story — a smarter way to organise memory, not a bigger computer. The related WorldParticle AI model from MIT is another recent example of one system doing more with less.

Should you care?

✅ Worth following, not yet worth switching to. If you build or study AI systems, Muir is one of the most concrete linear-attention results of 2026 — the 10-million-token and 25MB numbers are genuinely striking. But it is an unreviewed preprint at research scale with no public release. Watch for peer review and a weights release; don't plan a product around it today.

FAQs

What is linear attention?

Linear attention is a variant of the Transformer's attention mechanism that compresses everything the model has read into a fixed-size memory state, instead of storing every token separately. Computation grows linearly with input length rather than quadratically, and inference memory stays constant — which is why Muir can read 10 million tokens in 25MB.

How is Muir different from a Transformer?

A Transformer keeps a growing KV cache of every past token, so memory and per-step cost rise with context length. Muir replaces that with a fixed-size gated state, adds sensitivity-allocated low-rank gating to spend parameters efficiently, and uses a matched-filter readout so loud memories don't drown out distant ones. The result is constant memory with, the authors report, better recall than previous linear models.

What does reading 10 million tokens actually mean?

Ten million tokens is roughly 7–8 million English words — about a hundred full-length novels, or years of chat history. Muir was trained only on 4,096-token sequences and still held its prediction quality out to that length, which is the 2,560-times extrapolation the paper reports. It is a stress test, not a typical use case.

Can I use Muir today?

No. The paper is a preprint posted on 5 November 2026 and has not been peer-reviewed, and the authors have not released model weights or code. Google has not said whether Muir will appear in any product. If you want to follow it, the arXiv page linked in Sources below is the place to watch for updates.

TAKEAWAY: Muir shows that a smarter memory — not a bigger one — can let a small model read 10 million tokens in 25MB and still remember the details. The numbers are author-reported and unreviewed, so treat them as a promising lead, not a settled fact.

If you work with long documents or codebases, the paper's comparison table is the fastest way to see where linear attention wins — the arXiv link is in Sources below, and our agentic AI explainer shows what these long-context models are being built to do.

TechByCountry — Daily Tech News, Country by Country

Search This Blog