Llama 3.3 70B GPU Memory Calculator

What changed in Llama 3.3, and what did not

Meta published Llama 3.3 70B Instruct in December 2024. It is a text-only, instruction-tuned model, and Meta released no separate base checkpoint for it. Its improvements came from post-training, not from a new architecture: the configuration matches Llama 3.1 70B, with about 70.6B parameters, 80 layers, 64 query heads, 8 key-value heads and a 128K-token context. It officially supports eight languages (English, German, French, Italian, Portuguese, Hindi, Spanish and Thai) and ships under the Llama 3.3 Community License.

For capacity planning that is good news: anything you measured or sized for Llama 3.1 70B carries over unchanged. Swapping 3.1 Instruct for 3.3 is a model-file change, not a hardware change. This page therefore focuses on the questions people deploying 3.3 most often ask: how small a machine can run it, and what the quantized files really weigh.

How the number on the right is produced

Memory = weights + attention cache + a fixed allowance. Weights are 70 billion × bytes per parameter. The cache allowance is a flat 0.5 MB per cached token at 16-bit (0.25 MB at lower precisions) multiplied by context length and the number of simultaneous requests. The fixed allowance is 1.8 GB. The calculator then divides by 92% of the GPU size you enter to get a card count.

The default case, worked through

With 16-bit weights, an 8,192-token window, 2 concurrent requests and 80 GB cards, the result is 140.0 GB of weights, 8.0 GB of cache and 1.8 GB of overhead, 149.8 GB in total, or 3 × 80 GB GPUs (149.8 ÷ 73.6 = 2.04, rounded up). In deployment you would round to four GPUs, because tensor parallelism needs the 64 attention heads to divide evenly.

Most people running 3.3 do not use 16-bit weights, so here are the same inputs at other precisions:

The right-hand column differs because llama.cpp's Q4_K_M keeps some tensors at higher precision and averages close to 4.8 bits per weight, and because 4-bit GPU formats store group scales and often leave the embedding and output layers in 16-bit. When the margin is tight, check the actual file size on the model page instead of trusting 4.5 or 4 bits.

Small-machine options

• One 48 GB card (for example RTX 6000 Ada or A6000): a 4-bit build at one request and 8K context comes to 38.8 GB in the calculator, which fits with some cache headroom.
• Two 24 GB cards (RTX 3090 or 4090): the same 4-bit build splits across both. Consumer cards have no NVLink on the 4090, so layers are usually split sequentially, which works but is slower than one large card.
• Apple silicon or other unified-memory machines with 64 GB or more can run Q4 files through llama.cpp or MLX. Memory bandwidth there is far lower than an H100's, so generation is slower.
• One H100 80 GB at FP8: weights fit, but only a few GB remain for cache. Two GPUs, or an H200, is the comfortable FP8 configuration.

A bandwidth sanity check on speed

Each generated token reads every weight once. An H100 SXM5 moves 3.35 TB/s, so 70 GB of FP8 weights sets a single-stream ceiling near 48 tokens per second on one card, before any overhead. Two concurrent requests share those reads, so total throughput can be higher. The calculator's "tokens/second" line is a rough heuristic and can exceed this physical ceiling for one stream; do not quote it as a measured speed.

Where this estimate falls short

• Real BF16 cache for this architecture is about 0.31 MiB per token, so the 0.5 MB rule overstates long-context memory by roughly 60%.
• Memory for multimodal input does not apply, since 3.3 is text-only, but tool-calling prompts and long system prompts do count toward context.
• The status line and the GPU count use different headroom (95% and 92%); near the boundary, trust the GPU count.