Qwen3, Sparse Experts, and the Budget Behind a Token

An expert is a learned transformation

The 2021 Switch Transformers paper established an influential approach to scaling model capacity through sparse expert selection. Instead of applying every feed-forward subnetwork at every position, a learned router sends an input representation to selected experts. This historical source helps explain the concept, rather than supplying a performance forecast for a 2026 model. Experts are collections of trained numerical operations. They need not correspond neatly to human categories such as algebra, French, or software design, even when patterns of specialization emerge.

Suppose a fictional layer contains sixteen experts with equal arithmetic costs and chooses two for each token. At that layer, selected expert computation uses one eighth of the inventory. It does not follow that the complete model becomes eight times faster. Attention, routing, normalization, communication, and other layers remain. If those other tasks previously consumed half the time, even eliminating the expert computation entirely would improve speed by no more than a factor of two. This simple bound is a useful defense against turning a sparsity ratio into a latency guarantee.

Two model sizes describe two different resources

The Qwen3 technical report was submitted on May 14, 2025. Its flagship sparse model has 235 billion total parameters and 22 billion activated parameters per token. The report specifies 128 experts with eight selected at each expert layer and excludes shared experts in that particular design. It also describes dense members of the family. This matters because comparisons across the family are not comparisons of expert count alone: architecture, capacity, and training choices interact.

For a storage illustration, imagine placing 235 billion weights in a uniform two-byte representation. The raw weight payload would be 470 billion bytes, before caches, metadata, and runtime allocations. A similarly simplified 22-billion-weight payload would be 44 billion bytes. The first estimate is relevant to storing the full inventory; the second is not a substitute for it merely because only some weights are active. Real deployments can shard weights or use lower precision, but those are additional decisions whose transfer costs and quality implications need to be measured.

Sparse experts meet hybrid attention

The official Qwen3-Next model card, released in September 2025, describes an 80-billion-parameter model with three billion activated parameters. It lists 512 experts, ten activated experts, and one shared expert. Its attention layout alternates several Gated DeltaNet layers with gated attention. Those changes address different expenses: expert sparsity limits selected feed-forward work, while the hybrid sequence architecture changes how historical information is processed and retained. Calling the whole improvement simply a mixture-of-experts gain would hide that distinction.

A useful hypothetical benchmark would hold prompt length fixed while varying batch size, then hold batch size fixed while varying prompt length. If gains appear mainly as context grows, attention-state processing may be responsible. If gains depend strongly on the number of simultaneous requests, expert utilization and matrix shapes may matter more. A shared expert also contributes work on every routed token, so counting only the selected specialized experts omits part of the layer. Architectural diagrams become useful when they help design measurements, rather than serving as decorative evidence that a model must be efficient.

A current model is not the original Qwen3 checkpoint

Qwen’s official repository records a February 16, 2026 Qwen3.5 release and an August 12 release of Qwen3.8-2.4T-A95B. The latter model card identifies 2.4 trillion total parameters and 95 billion activated parameters, with a hybrid layout of Gated DeltaNet and gated attention layers. These are verifiable 2026 developments. They show continued investment in sparse conditional capacity, but they do not retroactively change the architecture or benchmarks of the May 2025 model.

The current card also distinguishes the released weight artifact from a managed service with additional features. That distinction affects reproducibility. A benchmark run against a hosted assistant can include tools, settings, and context handling that are absent from a downloaded checkpoint. My interpretation is that a useful model comparison should identify the actual artifact, serving stack, and supported operating mode before discussing quality or price. A family name is a research lineage, not a sufficiently precise experimental specification.

A cheap step can still make an expensive answer

The original Qwen3 report integrates thinking and non-thinking behavior and studies a controllable thinking budget. The newer Qwen3.8 card exposes reasoning-effort controls. These controls operate on a different axis from expert sparsity. A model can spend less arithmetic per generated step yet produce many more steps before answering. That additional computation can improve some tasks, but it remains part of the service’s latency and resource bill.

Consider two imaginary systems with identical prompt processing time. System A takes 20 milliseconds per generated token and produces 500 tokens; system B takes 10 milliseconds and produces 2,000. Their generation times are ten and twenty seconds respectively, ignoring batching and network delay. B has the faster step and the slower answer. This arithmetic does not describe any Qwen measurement. It explains why evaluations should report answer quality against both elapsed time and total tokens, including hidden or separately returned reasoning, instead of comparing only peak decoding throughput.

Compare completed work under the same constraint

My proposed deployment test would begin with a fixed set of short questions, long documents, and multi-step coding tasks. It would measure successful answers per unit of wall-clock time and per available memory budget. Each configuration would disclose precision, hardware, concurrency, context length, reasoning limit, and any tools. Runs should include a cold start and a sustained workload, because distributing expert weights and warming kernels can affect the experience differently from steady-state arithmetic.

Several limitations should remain visible. Public benchmark tables are measurements on selected tasks, not guarantees of reliability across unfamiliar work. A larger expert inventory can increase deployment complexity. Lower average activation does not eliminate worst-case routing congestion. New attention layouts may change failure patterns on distant evidence. The strongest conclusion supported by the Qwen progression is that capacity and computation can be separated more flexibly than in a dense model. Whether that flexibility makes a particular application better is an empirical question answered by complete jobs, with all of their tokens and waiting time counted.

A particularly informative failure case would be a mixed queue of easy requests and expensive reasoning jobs. If the long jobs monopolize resources, average throughput can rise while interactive users wait longer. I would therefore publish median and upper-tail response times separately, and count abandoned or timed-out requests. Resource efficiency should improve access to completed answers, not merely the count of internal operations performed while users are waiting.