GPU Sizing for MoE Models: Why 400B Total Isn't 400B of VRAM

If you have priced out GPUs for a self-hosted deployment lately, the parameter counts on the latest open-weight models can be alarming. DeepSeek V3 ships as 671B parameters. Llama 4 Maverick is 400B. Qwen3 tops out at 235B. Read those numbers the way you read a dense model and you will conclude you need a small cluster just to load the weights. But that instinct is wrong, and the reason is architecture: these are all Mixture-of-Experts (MoE) models. The headline parameter count and the number that actually does work on each token are two very different things.

Getting this distinction right is the single most useful thing you can do when sizing hardware for on-prem inference. It changes both what you buy and what performance you should expect from it.

Total vs. active parameters

A dense model runs every parameter through every token. A MoE model instead splits much of the network into many experts and, for each token, a router selects only a small handful to activate. The rest sit idle for that token. This gives you two separate numbers:

• Total parameters — every expert's weights, all of which must be loaded somewhere. This is the big, scary number.
• Active parameters — the subset that actually fires per token. This is the number that governs how much math happens.

The current flagships make the gap dramatic:

• Llama 4 Maverick — 400B total, 17B active
• Llama 4 Scout — 109B total, 17B active
• Qwen3-235B-A22B — 235B total, 22B active
• DeepSeek V3 / R1 — 671B total, 37B active

The rule that makes sizing simple

Once you internalize where each number lands, capacity planning falls into place around a single split:

Why does memory follow the total? Because the router can pick any expert on any token, so every expert's weights must be resident in GPU memory at all times. You cannot page them in on demand without wrecking latency. All 400B parameters occupy VRAM even though only 17B participate in a given forward pass.

Why does speed follow the active count? Because the actual matrix multiplies for a token only touch the selected experts. The FLOPs per token scale with active parameters, not total. That is precisely the trade MoE is designed to buy: the quality headroom of a very large model at close to the inference cost of a small one — as long as you can afford to keep the whole thing in memory.

Turning parameters into gigabytes

To estimate VRAM, start with the weights. The rules of thumb are precision-dependent, and they are planning approximations rather than guarantees:

• BF16 / FP16 — roughly 2 bytes per parameter
• 8-bit (FP8 / INT8) — roughly 1 byte per parameter
• 4-bit — roughly 0.5 byte per parameter

Multiply by the total parameter count and you have the weight footprint. But weights are not the whole story. Add:

• KV cache — grows with context length and batch size. Long-context serving can consume a serious slice of memory, and Llama 4's very large context windows make this line item especially important to budget for.
• Activation and runtime overhead — working buffers, CUDA context, framework reservations. Leave headroom rather than sizing to the exact weight figure.

Worked sizing, qualitatively

Apply the weight rule to the flagships and the hardware picture separates cleanly. These are directional, not promises — your context length, batch size, and quantization scheme move the lines:

• Scout (109B total) — at 4-bit, the weights compress enough to fit on a single 80GB GPU, leaving room for a modest KV cache. This is the friendliest of the group to deploy.
• Maverick (400B total) — realistically a multi-GPU node, or heavy quantization if you are trying to squeeze the footprint down. No single accelerator holds 400B comfortably.
• DeepSeek V3 / R1 (671B total) — a multi-GPU server, full stop. The weights alone dominate any single card.

The crucial payoff: despite those memory demands, throughput stays high. Maverick activates only 17B and DeepSeek only 37B per token, so once the weights are resident, these models generate tokens far faster than their total size suggests. You are paying for memory, not for a proportional slowdown in speed.

Takeaway table

Practical guidance

Pulling it together into decisions you can act on:

• Quantize to fit. FP8, AWQ, or INT4 are the levers that move a model from "needs a cluster" to "fits the node you have." The step from BF16 to 4-bit roughly quarters the weight footprint.
• Budget the KV cache explicitly. If you plan to serve long contexts or large batches, reserve memory for it up front rather than discovering the ceiling in production.
• Size two dimensions independently. Pick the active-parameter class that meets your latency target, and pick the total-parameter count plus quantization that fits your VRAM budget. They are separate knobs — MoE is what lets you turn them separately.

Where this lands for on-prem

The reason MoE matters for self-hosting is that it makes right-sizing both more important and more rewarding. Read the parameter counts naively and you over-provision, buying compute you will never use to serve a model that only activates a fraction of itself. Read them correctly and you can run a frontier-class open-weight model on hardware that is far more modest than the headline number implies — high throughput from the small active set, memory sized to the total, quantization bridging the gap.

That calculation — matching a specific model, quantization, and context budget to the smallest hardware that will actually serve it well — is exactly the work LLMDeploy does when it stands up open-weight LLMs on your own infrastructure. The 400B on the model card is not 400B of VRAM, and knowing the difference is the first step to a deployment that is both capable and affordable.