DeepSeek-V3 (671B) GPU Memory Calculator

What DeepSeek-V3 actually is

DeepSeek-V3 is an open-weight mixture-of-experts (MoE) language model released by DeepSeek in December 2024. It has 671 billion parameters in total, but only about 37 billion are active for any single token: each of its 61 transformer layers (apart from the first three, which are dense) routes a token to 8 of 256 "routed" experts plus one shared expert. The published context window is 128K tokens. DeepSeek reports pre-training on 14.8 trillion tokens and, unusually for an open model, training in FP8 mixed precision. The checkpoint on Hugging Face is about 685B parameters because it also contains a 14B multi-token-prediction module that ordinary inference does not need.

Two design choices matter most for memory. First, the MoE layout means compute per token resembles a 37B dense model, while storage still has to hold all 671B parameters: every expert must sit in GPU memory (or be offloaded, at a large speed cost) because the router can pick any of them. Second, DeepSeek uses Multi-head Latent Attention (MLA), which stores a compressed 512-value latent plus a 64-value rotary key per layer instead of full keys and values for every head. That makes the real KV cache very small, as shown below.

How this calculator estimates memory

The calculator adds three numbers:

The weights term is exact arithmetic: 671 × 10⁹ parameters × 2 bytes is 1,342 GB. The KV term is a rule of thumb that scales a "0.5 MB per token for a 70B model" figure with parameter count. For DeepSeek-V3 that rule gives about 4.8 MB per token, which is far from the architecture. With MLA the cache is 61 layers × 576 values × 2 bytes ≈ 69 KiB per token in BF16, roughly seventy times smaller, provided your inference engine keeps the compressed latent form (SGLang and recent vLLM releases do; naive implementations that expand it to full heads do not). Treat the calculator's KV line as a generous upper bound for this model.

Worked example: the default inputs

With the page defaults (671B parameters, FP16/BF16, 8,192-token context, batch of 2, 80 GB per GPU) the calculator reports:

Using the MLA figure instead, the cache for the same 16,384 tokens would be about 1.1 GB, so the total is dominated almost entirely by weights. Either way, BF16 DeepSeek-V3 does not fit in one 8-GPU H100 server (8 × 80 GB = 640 GB). Twenty GPUs is also not a layout engines accept for tensor parallelism; in practice you would use three 8-GPU nodes with pipeline or expert parallelism, or more realistically not run it in BF16 at all.

Choosing a precision for DeepSeek-V3

Because the released weights are already FP8, the 16-bit setting describes an up-converted copy, mainly useful on hardware without FP8 support (for example A100). Switching the calculator's precision gives these totals at the default context and batch:

• FP8 on one 8 × H200 node (8 × 141 GB = 1,128 GB) is the single-node layout SGLang's DeepSeek documentation uses as its example. It holds the native weights with several hundred GB left for KV cache and batching.
• FP8 on 8 × H100 does not fit: 671 GB of weights exceeds 640 GB before any cache. You need two H100 nodes connected by InfiniBand or RoCE, which brings inter-node latency into every token.
• 4-bit community quantizations bring the weights into the 340–380 GB range, which fits one 8 × H100 node or a few large-memory workstations. Expect some quality loss; measure it on your own tasks rather than trusting a single perplexity number.
• CPU offload of experts (llama.cpp, KTransformers) lets machines with a lot of system RAM run the model slowly, since only ~37B parameters are touched per token. This is a hobbyist path, not a serving setup.

Why speed does not follow from memory here

For a dense model, decoding one token reads every weight once, so single-stream speed is bounded by memory bandwidth divided by model size. For DeepSeek-V3 each token reads only the shared parts plus 8 experts, about 37B parameters, so the ceiling is much higher than the 671B total suggests. The calculator's "tokens per second" line uses a simple formula based on total parameters and will look pessimistic for this model; it is not a measurement. Real throughput depends heavily on expert-parallel load balancing and on batch size.

Limitations of this estimate

• The KV rule ignores MLA and overstates DeepSeek-V3's cache by about 70×.
• Activation memory, CUDA graphs and the engine's own preallocation are folded into a flat 1.8 GB plus an 8% headroom factor. vLLM, for example, reserves a fixed fraction of GPU memory (90% by default) regardless of model size.
• The GPU count is a memory floor, not a valid parallel layout. Check that your engine supports the tensor, pipeline and expert parallel sizes you plan to use.
• The multi-token-prediction module (about 14B parameters) is excluded; add roughly 28 GB in BF16 if you load it for speculative decoding.