The first few messages are fast. An hour into a long session, with a big document pasted in and a coding agent halfway through a refactor, the same model feels sluggish.

The usual explanation is that the model is "thinking harder". It is not.

The model does exactly the same amount of work per word at message one and at message ten thousand. What changed is how much it has to read before it can produce each word.

This article is about what it reads, why that grows, and what you can do about it. I will end with measurements from my own research, but most of what follows applies whatever stack you run.

What the model keeps in memory

When a language model generates text, it produces one token at a time. To produce the next token, it has to look back at everything that came before.

Recomputing that history on every token would be quadratically disastrous. So every serving stack keeps a cache instead.

For each token in the conversation, and for each layer of the model, it stores two vectors: a key and a value. This is the KV cache.

Here is the mental model that helps. The keys are an index, and the values are the content.

When the model generates a new token, it forms a query vector. It compares that query against every key in the cache to decide how relevant each past token is. Then it mixes the corresponding values together in proportion to that relevance.

That is attention. It happens in every layer, for every token generated.

So the cost of generating one token is dominated by one thing: reading the entire KV cache.

How big the cache actually gets

Take Qwen3-8B. It has 36 layers and 8 key-value heads per layer, each holding 128 numbers per token.

Each token needs a key and a value in every one of those heads, in every layer. At 16-bit precision that works out to about 144 KB of cache for every single token of context.

Multiply that out:

• 32,000 tokens of context: about 4.8 GB of cache, read once per generated token

• 1,000,000 tokens: about 150 GB

Now think about what that means physically.

An A100 has 80 GB of memory in total, and your model weights already sit in it. Long before you reach a million tokens, the cache stops fitting beside the weights.

It has to live somewhere slower. Usually that is ordinary system RAM, reached across a PCIe bus roughly an order of magnitude slower than the GPU's own memory.

This is what vLLM's CPU offload, LMCache and similar layers do. It is not exotic any more. If you run agents with long context, you are probably already here.

The first fix: stop attending to everything

Here is the observation that a lot of good work is built on.

When a model attends over a long context, the attention weights are extremely concentrated. A handful of tokens get almost all of the weight. The long tail contributes close to nothing.

So do not read the long tail. Score every key, take the top k, and compute attention over just those k tokens.

This is top-k sparse attention. With k of a few hundred to a few thousand, it stays remarkably faithful to full attention.

The savings look enormous. Instead of reading 150 GB, you read the values for k tokens. At k=512 that is a couple of hundred megabytes, no matter how long the conversation gets.

The cost stops growing with context. Except it does not, and this is the part that surprised me.

The catch

To take the top k keys, you have to score all n keys. You cannot know which ones matter without looking.

So every top-k method carries a second, cheaper data structure. It is a compressed representation of every key, small enough to scan quickly.

Loki keeps 32 principal components. Double Sparsity keeps 32 channels chosen offline. SparQ lets each query pick its own 32 channels.

In practice these all land around the same size, roughly 136 bits per token. That sounds tiny.

Run the numbers anyway. For Qwen3-8B, 136 bits per token across all 288 layer-and-head pairs comes to:

• about 160 MB per generated token at 32,000 tokens of context

• about 5.1 GB per generated token at 1,000,000 tokens

And unlike the k winners, this one grows linearly with the conversation.

So at long context, the thing your decode step waits on is not the model weights. It is not the tokens the model decided to attend to either.

It is the index scan used to decide.

In my measurements the winner rows moved about 200 MB per step while the scan moved 5.1 GB. Most of the effort in this area has gone into the 4%.

Where the memory lives changes the answer

Before going further, one distinction matters enormously. It is the thing I would most like developers to take away from this article.

If that index sits in GPU memory, reading fewer bytes barely helps. GPU memory is fast, and the scan is limited by arithmetic rather than by bandwidth. Reading less does not reduce the number of multiply-adds.

If the index sits in host memory, bytes are time, almost exactly. Everything moves at the speed of the bus. The arithmetic units sit idle waiting, and halving the bytes nearly halves the step.

Any efficiency claim you read about KV caches is implicitly about one of these two worlds. Check which one before you believe it applies to you.

How to read fewer bits per key

Given that, the lever in the offloaded world is obvious. Read fewer bits per key.

Every method above does this by choosing fewer channels. Keep 32 of the 128 channels, read them at 4 bits, discard the rest. The choice is made once, at design time.

The alternative is to keep all the channels but read each one to a different depth.

To see why that might work, you need one fact about quantization. When you store a number at 4 bits instead of 16, you introduce rounding error, and each additional bit cuts that error by a factor of four.

So precision has sharply diminishing returns. Going from 1 bit to 2 bits on a channel buys you a lot. Going from 3 bits to 4 buys you very little.

Now combine that with what a query actually needs.

For any given query, most of the score comes from a few dozen channels, and which ones differ from query to query. So the fourth bit of an important channel can genuinely be worth less than the first bit of a channel you were about to throw away.

That means there is an optimal allocation, and it is not "all channels at the same depth".

Bit planes: how a partial read becomes a valid read

The obstacle is storage.

If your cache is stored the ordinary way, as 4-bit numbers laid out one after another, you cannot read "the first two bits" of each number without reading the whole thing anyway. And what you get back is not a meaningful number.

The fix is to transpose the storage. Instead of storing numbers, store bit planes.

All the first bits of a block of tokens go together, then all the second bits, and so on.

ordinary layout:   [1011][0110][1101][0010] ...   four numbers, four bits each

bit-plane layout:  [1010...]  plane 0   the most significant bit of every number
                   [0110...]  plane 1
                   [1101...]  plane 2
                   [1010...]  plane 3

Now read only plane 0 and plane 1.

What you have is every number rounded to 2 bits. Not an approximation of one, and not garbage. Exactly the 2-bit version, with the same scale factor.

Reading a prefix of the planes is mathematically identical to having stored a lower-precision copy.

This idea is not mine. Any-Precision LLM used it for model weights, so that a prefix read gives you a smaller model.

What is new here is the application to the key cache, so that a prefix read gives you a cheaper index. And then choosing the prefix length per query.

How each query picks its own depth

With the storage sorted out, the allocation problem has a clean answer.

For a given query, each channel has an importance, which is roughly how much of the score variance that channel carries.

Spending one more bit on a channel reduces the remaining error by an amount proportional to that importance, divided by four to the power of the bits already spent.

Allocating a fixed budget to maximise the total reduction is a classic problem with a closed form. It is called reverse water-filling.

You pick a threshold. You give every channel however many bits puts it at that threshold. Then you binary search the threshold until the total hits your budget.

Important channels get all four planes. Marginal channels get one. Channels that carry nothing for this query get skipped entirely.

That is the whole method. Store the 4-bit key cache as bit planes, and let each query water-fill its bit budget across channels.

What the measurements show

I measured this on rented A100s across five models, from Llama-3.1-8B to Qwen2.5-7B-Instruct-1M, at context lengths up to 128,000 tokens. I timed full decode steps out to a million.

Accuracy per byte

To match the attention error of the 136-bit fixed-depth scans, this needs 46 to 74 bits per token. That holds across all seven model and context settings tested.

So the scan reads 1.8 to 2.9 times fewer bytes at the same quality.

End-to-end speed

With the cache and index in pinned host memory on an A100, a decode step at one million tokens runs 1.67 times faster in GPU time than with the 136-bit scans. It is 2.5 times faster than with a block-landmark index.

Downstream behaviour

On synthetic long-context retrieval tasks, every per-token scan matches exact top-k decoding. Those tasks do not separate the methods at all.

On real coding-agent sessions of 80,000 to 100,000 tokens, decoding the agent's next step, the gap is visible. At a 2% attention budget this agrees with exact top-k decoding 0.67 of the time, against 0.49 for the closest comparable method.

Where this does not work

Back to the distinction from earlier, because this is the honest part.

Move the index into GPU memory and the method is not faster. It is slower.

Extracting individual bits costs one or two integer operations per bit, roughly four times the per-bit cost of reading a 4-bit number whole. The kernel ends up about 1.4 times slower than a plain 4-bit scan, despite reading 38% fewer bytes.

That is not a quirk of one GPU.

Counting integer lanes against memory bandwidth, an A100 offers roughly 5 integer operations per byte of bandwidth. An H100 is about the same. If a scan is arithmetic-bound on one current data-centre GPU, it is arithmetic-bound on all of them.

There is also a storage cost. As a separate index this is 68 bytes per token per KV head, four times what Double Sparsity's label cache costs.

It only becomes free in a stack whose key cache is already 4-bit and stored channel-major, where the index and the cache are the same bytes.

One measurement worth carrying around

Even if you never touch any of this, keep this number.

Copying from pinned host memory in contiguous runs reached about 26 GB/s at every size from 4 KB upward. Issuing one copy call per run instead got 0.4 GB/s at 4 KB.

Same bytes. Same hardware. Sixty times slower, purely because of how the reads were grouped.

Layout is not a detail.

What to do with this

If you run long-context inference today, three things here are worth more to you than the method itself.

Find out whether your KV index lives in GPU memory or in host memory. That single fact decides whether any efficiency claim you read applies to you at all.

Measure GPU time, not wall-clock, when your harness is Python. In my own measurements the two disagreed by up to 25%, and the gap differed by method, which is exactly how you end up reporting the wrong winner.

Look at your memory layout before you look at your kernel. The 26 GB/s against 0.4 GB/s result above was not a code change. It was the same bytes, grouped differently.

If you want the full method, the measurements, or the code to check any number in this article yourself, it is all at github.com/vivekkalyanarangan30/fathom.

Paper: https://arxiv.org/abs/2609.17652