Shrinking the Key–Value Cache Without Losing the Question

A cache has a different scaling problem from weights

Model weights are mostly fixed during an inference request. Cache state grows as the request progresses. As a purely illustrative calculation, assume a model stores 1,024 key and value numbers combined per token at each of 32 layers, using two bytes per number. A 100,000-token sequence then consumes 6.5536 billion bytes for that state. Eight independent sequences would require about 52.4 billion bytes before overhead. These dimensions are hypothetical, but the multiplication explains why a server can fit a model and still run out of room for its conversations.

There are several ways to change the multiplication. Quantization uses fewer bits per stored number. Eviction retains fewer token states. Feature compression stores shorter representations. Architectural changes may avoid maintaining a full token history in some layers. These mechanisms are not interchangeable. Discarding a token removes a particular item, whereas rounding all keys perturbs matching scores throughout the sequence. The appropriate evaluation must reveal which mechanism was applied, whether training was required, and whether the stated compression includes the metadata and scales needed to use the result.

Compressing a key changes where attention looks

TurboQuant, submitted in April 2025, studies online vector quantization through randomized rotations and scalar quantizers, with a residual correction for inner-product estimation. Its theoretical claims concern distortion under stated conditions, and its empirical cache results concern the models and tasks tested. The connection to attention is direct: query–key inner products determine which earlier states receive weight. A small geometric error can matter differently depending on whether it changes the ranking of competing evidence.

Imagine two candidate passages whose query scores are 5.00 and 4.99. A perturbation of 0.02 could reverse their order. The same perturbation applied to scores 5.00 and 1.00 would preserve a wide separation. This invented example does not model the complete softmax calculation, but it illustrates why an average vector-reconstruction error is not a complete prediction of answer quality. Evidence may be redundant, or a single narrow distinction may decide the answer. Distortion needs to be interpreted through the operation that consumes the compressed values.

Spend bits where the output is sensitive

The August 14, 2026 preprint on attention-aware transform coding formalizes one version of that idea. Under a white-noise quantization model, its analysis separates attention-aware distortion into key and value contributions across tokens and channels. The proposed allocation uses calibration data to assign bits to the directions where error matters. The authors report approximately 5.8-fold compression with near-preserved accuracy on their selected models and benchmark suite. That result is conditional on the tested setup and the approximation behind the distortion model.

A useful independent analogy is an image that will later be used to read a tiny serial number. Allocating bits uniformly across the photograph may waste precision on a blank wall while damaging the characters. A task-aware scheme would protect the informative region. In a language cache the informative direction is less visible and can change with the query, making calibration coverage important. My inference is that such methods should be tested under deliberate distribution shifts: unfamiliar document formats, new instruction styles, and questions that depend on information rarely used in calibration.

An October 2026 result joins token and feature compression

SlimKV, submitted on October 2, 2026 and listed as accepted to EMNLP 2026, combines token-level condensation with low-rank latent cache representations. Its learned beacon states summarize context, and its design avoids reconstructing full-dimensional beacon keys during decoding. The paper reports strong results at several compression ratios and end-to-end decoding improvements in its long-context tests. Its method requires training adjustments, so it should not be compared to a training-free quantizer as if their adoption costs were identical.

The distinction between attention-kernel speed and whole-model speed is especially useful here. Suppose a fictional service spends 40 percent of decoding time inside attention and 60 percent elsewhere. Making attention four times faster changes total time from one unit to 0.1 plus 0.6 units, an overall gain of about 1.43 times. This is not a SlimKV measurement. It shows why an impressive kernel improvement can coexist with a smaller end-to-end improvement, and why both should be reported without treating the smaller ratio as a failure.

A retained token may still be functionally unavailable

The Pitfalls of KV Cache Compression, revised in May 2026 and identified as an ACL 2026 paper, evaluates several eviction methods under multiple simultaneous instructions. Its experiments show uneven degradation across instruction types and examine system-prompt leakage. The result does not establish that every compressed system leaks prompts. It demonstrates that a favorable score on a narrow retrieval task is insufficient evidence about instruction retention in a different workload.

A March 2026 preprint on attention dynamics separately distinguishes keeping information from making it reachable and actually using it. Its synthetic tasks probe tracking, disambiguation, and multi-hop reasoning under compression. The authors report sharp deterioration at high compression in their tests; that is not a universal threshold to transfer across all architectures. These studies suggest a practical stress test: put a formatting rule early in the context, introduce several similar entities later, then ask a question requiring both the rule and the correct entity relationship. Success requires several forms of memory to cooperate.

Measure useful capacity, with a recovery path

My proposed comparison would pair each memory ratio with latency, throughput, and a breakdown of errors. It would test several lengths and evidence positions, including repeated follow-up questions that differ from the first query. Compression and decompression time must be counted. For a learned method, the training budget and compatibility constraints belong in the report. For a calibration-based method, the calibration set and its relation to the evaluation distribution should be explicit.

There is also an application-level design question. If a cheaper cache fails a confidence or consistency check, can the service re-read the original document using a less compressed configuration? Such a fallback costs time, but it may be preferable to silently discarding a crucial condition. This is a proposed design choice, not a result demonstrated by all the cited papers. The emerging 2026 lesson is that cache capacity should be described in terms of usable evidence and preserved behavior. Saving memory helps only when the model can still answer the question that arrives afterward.

Repeated questioning deserves its own test. A document summary can answer the first broad question well and lose a minor exception needed by the fifth follow-up. I would keep the original prompt fixed, vary the order of later questions, and compare failures against the uncompressed baseline. That design helps distinguish information lost during compression from information the original model could never retrieve reliably in the first place.