Exact VRAM memory allocation, dynamic KV-cache requirements, and tensor parallelism slicing for Qwen-2.5 7B Consumer-Grade quantized in FP16 Uncompressed Native deployed on NVIDIA H100 80GB SXM5.
Yes. BF16 needs about 17.9 GB at the default settings, which fits a 24 GB card. 8-bit (9.8 GB) fits 12–16 GB cards, and 4-bit fits 8 GB cards with short contexts.
Very close. The real cache is 56 KiB per token in BF16 (28 layers, 4 KV heads), and the calculator's scaled figure is about 55.6 KiB.
It is released under the Apache 2.0 licence. Read the licence text on the model card for the exact terms.
Qwen2.5-7B is the 7-billion-class member of the Qwen2.5 series released in September 2024. The model card gives 7.61 billion parameters (6.53 billion excluding embeddings), 28 layers, and grouped-query attention with 28 query heads and only 4 key-value heads. Context length is 131,072 tokens (with YaRN enabled beyond the configured 32,768) and generation up to 8,192 tokens. It is licensed under Apache 2.0, which makes it one of the easiest models in this size class to use commercially. The coding and maths specialists, Qwen2.5-Coder-7B and Qwen2.5-Math-7B, are built on the same base model, so the memory figures here apply to them as well.
This calculator's default target is an H100 80 GB, which is far more memory than a 7B model needs. The useful questions are the reverse of the large-model pages: how small a GPU will do, and how many users one large GPU can serve.
Weights are 7.6 × bytes per parameter. The cache per token is the calculator's 70B reference (0.5 MB) scaled by 7.6 ÷ 70, about 55.6 KiB at 16-bit. The model's actual cache is 2 × 28 layers × 4 KV heads × 128 dimensions × 2 bytes = 57,344 bytes, or 56 KiB per token. The two agree to within 1%, largely by coincidence, since the small number of KV heads offsets the scaling rule's bias. Add 1.8 GB for the CUDA runtime and divide by 92% of the card.
At FP16/BF16, 8,192 tokens, 2 concurrent requests and an 80 GB card: weights 15.2 GB, cache 0.87 GB, overhead 1.8 GB, for 17.9 GB in total. The status shows it fits on one GPU. Nearly 60 GB of the H100 is left unused.
That spare memory is the real resource. After weights and overhead, 80 × 0.92 − 15.2 − 1.8 leaves about 56.6 GB, and at 56 KiB per token that holds roughly 980,000 cached tokens. In practice that is on the order of a hundred concurrent 8K-token conversations on a single card, which is why serving engines with paged KV cache can run small models at high concurrency. Try batch 16 at 8-bit in the calculator: 12.9 GB in total.
| GPU memory | FP16/BF16 (17.9 GB at 8K × 2) | 8-bit (9.8 GB) | 4-bit (6.0 GB) |
|---|---|---|---|
| 8 GB (e.g. RTX 4060) | No | No | Yes, short contexts |
| 12 GB (e.g. RTX 3060 12 GB) | No | Yes | Yes |
| 16 GB (e.g. RTX 4060 Ti 16 GB) | No | Yes | Yes |
| 24 GB (RTX 3090 / 4090) | Yes | Yes | Yes, with long contexts |
| 80 GB (H100) | Yes, with many users | Yes | Rarely worth it |
"Yes" means the calculator total is under 95% of the card. A real Q4_K_M GGUF of this model is about 4.7 GB, slightly larger than the calculator's 3.8 GB 4-bit weight figure, because some tensors stay at higher precision. Desktop GPUs also share memory with your display, so keep a gigabyte or so spare.
On an H100 SXM5 (3.35 TB/s), streaming 15.2 GB of BF16 weights per token limits a single stream to about 220 tokens per second at best. On an RTX 4090 (about 1 TB/s) the same limit is about 66 tokens per second at BF16, or about twice that with 8-bit weights. These are ceilings from memory bandwidth, not measured speeds. The calculator's own "~368 tokens/second" for the default case is a heuristic for two concurrent requests and should be read as an order of magnitude only.