ARTIFICIAL INTELLIGENCE
DeepSeek-V3 and the Cost of Remembering a Conversation
Why latent attention and balanced expert routing still matter in 2026, with worked memory examples and a careful reading of DeepSeek’s reported costs.
Research checked
Newest cited publication
An attention-head schematic introduces one source of language-model computation and memory traffic; DeepSeek-V3-specific changes are explained in the article.
Cosmia Nebula · Image source · CC BY-SA 4.0
A language model can finish reading a document and still face an expensive problem: keeping enough of that document available while it writes an answer. Every new token asks earlier tokens what matters. DeepSeek-V3 is a useful case study because it tackles that memory burden alongside a separate problem, distributing work among expert networks. Its original report appeared on December 27, 2024, with a February 18, 2025 revision. It is a historical baseline, not a claim about the newest DeepSeek release. The April 2026 V4 announcement and May 2026 research on expert balancing show why the underlying questions remain current. Understanding them requires keeping three budgets separate: arithmetic, stored state, and communication between devices. A reduction in one can leave another almost untouched.
What a smaller memory actually stores
In ordinary attention, each earlier token contributes a key for matching future questions and a value containing information to retrieve. Multi-head latent attention, introduced in DeepSeek-V2, replaces much of the separately stored key and value representation with a smaller learned latent vector. Appropriate projections recover the information needed by the attention calculation. Position information needs special handling, so the cache also retains a decoupled positional component. This is an architectural decision learned during training, rather than a lossless compression utility that can be attached to any existing model without consequences.
A deliberately simplified calculation shows the attraction. Suppose a hypothetical layer otherwise stores 4,096 numbers per token, each occupying two bytes. A 100,000-token conversation then requires 819.2 million bytes for that layer alone. Replacing that representation with 640 stored numbers would bring the same calculation to 128 million bytes. These are illustrative dimensions, not DeepSeek configuration values. The ratio is 6.4 to one, before allocation overhead, batching, and other memory. Multiplying either requirement by dozens of layers explains why attention memory can dominate a server even after its weights have been loaded.
Sparse computation still needs a destination
V3 combines latent attention with a mixture of experts. Its report describes 671 billion total parameters with roughly 37 billion activated per token. An expert is a learned feed-forward subnetwork; a router chooses a small subset for each token. The large parameter inventory provides conditional capacity while the selected subset limits arithmetic. That does not mean the whole model fits in the memory required by a dense 37-billion-parameter network. In a typical deployment, inactive experts still have to reside somewhere accessible, and routing can require data movement across devices.
Consider an imaginary service with eight expert workers and 800 assignments in a scheduling interval. Perfect balance gives each worker 100 assignments. If one receives 300, the interval can finish only when that worker completes its larger queue, even if several others become idle. The exact delay depends on overlap and capacity, but the counting alone shows why average computation is a poor predictor of completion time. Sparse models therefore need both competent decisions about which experts to consult and a way to prevent the consultation pattern from overwhelming a few destinations.
Balancing work without redefining the language task
The auxiliary-loss-free balancing paper, published in August 2024, proposes changing expert-specific routing biases using recent workloads. Overused experts become less attractive to the selection procedure; underused ones become more attractive. The objective is to regulate traffic without injecting the same extra balancing gradients into the main language-learning objective. Its reported tests extended to models with three billion parameters and 200 billion training tokens. Those experiments establish evidence at their tested scales, rather than a universal theorem that every bias update is safe.
A helpful engineering analogy is adjusting entry prices to several equally useful queues. Prices alter where new arrivals go, while the work performed inside each queue remains defined by the original task. The analogy has a limit: learned experts are not interchangeable checkout counters. Steering a token away from a preferred expert may affect its representation. A sensible evaluation must therefore examine predictive quality together with load distribution. Reporting only an impressively flat utilization histogram would miss the central tradeoff that motivated the method.
The balancing question remains open
A May 14, 2026 preprint on phi-balancing targets expected expert utilization across the data distribution. Its authors distinguish that population objective from the noisy assignments observed within individual minibatches. They derive an online adjustment using convex duality and mirror descent, reporting improved expert use in their pretraining and fine-tuning experiments. That is a research claim under the paper’s assumptions and comparisons, not evidence that V3 secretly used the later method or that a single router has won permanently.
The conceptual issue is easy to see with two types of documents. One batch may contain mostly code and legitimately favor a subset of experts; another may contain ordinary prose. Forcing both batches to look identical could suppress useful specialization. Ignoring imbalance altogether could starve other experts of training opportunities. Population balancing asks how to regulate the long-run traffic without erasing short-run structure. My inference is that routing should be treated as a control problem with a time horizon, not merely a penalty coefficient selected once at the beginning of a run.
A training bill needs an accounting boundary
V3 reports 2.788 million H800 GPU hours for its official training stages. Its dollar conversion assumes a rental price, and the report excludes earlier research and ablation experiments from that figure. That accounting boundary is crucial. Training hours do not measure the cost of discovering the architecture, collecting data, operating a product, or serving later customer traffic. Nor can the reported total isolate the contribution of latent attention from expert routing and the many infrastructure changes applied together.
Imagine two teams that each consume one million accelerator hours in a final run. One reached that recipe on its first attempt; the other spent nine million hours on failed attempts. Their final-run efficiency is identical under that metric, while their total research expenditure differs tenfold. Both statements can be true. The useful comparison is to declare the boundary, hardware, precision, token budget, and final quality before comparing costs. A number becomes informative when another team knows what it would have to reproduce, including the parts outside the most attractive headline.
What the next measurement should settle
DeepSeek’s April 24, 2026 V4 preview announcement describes token compression combined with sparse attention for million-token context. The official release establishes that the family moved beyond the V3 baseline; it does not make a V3 cache estimate a measurement of V4. Different mechanisms may reduce the bytes retained, the tokens attended to, or the cost of moving those bytes. These gains should be measured separately before they are combined into an end-to-end claim.
For a practical comparison, I would request latency to first output, subsequent token latency, concurrent request capacity, memory at several context lengths, and accuracy on questions whose evidence appears far apart. The same experiment should include skewed expert traffic and short interactive prompts, where communication overhead can outweigh arithmetic savings. A smaller cache is valuable when it preserves the needed information and improves the service people actually use. The lasting lesson of V3 is the discipline of redesigning several budgets together, while refusing to let one favorable ratio stand in for the whole system.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelarXivPublished Checked
- DeepSeek-V3 Technical ReportarXivPublished Checked
- Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-ExpertsarXivPublished Checked
- ϕ-Balancing for Mixture-of-Experts TrainingarXivPublished Checked
- DeepSeek-V4 Preview: Entering the Era of Affordable Million-Token ContextDeepSeekPublished Checked
FOLLOW THE QUESTION
Study engineering foundations