Will It Fit?
Estimate single-GPU text inference VRAM across Transformers and vLLM with a compact, explainable breakdown.
GPT-OSS 20B · vLLM · Mixed MXFP4 + BF16 checkpoint
Required GPU VRAM (0.9 budget)
17.1 GB
Core estimate: 15.3 GB. Against RTX 4090 24GB, this leaves 8.7 GB of headroom.
At 4,096 tokens, the estimated max concurrency is 70 concurrent requests.
Quick read
Selected GPU
Runtime
KV cache dtype
Context length
Current concurrency
Max concurrency @ context
Class
Bandwidth
Nominal VRAM
Core estimate
Required GPU VRAM (0.9 budget)
Headroom
Breakdown
Where the memory goes
Weights
21B resident parameters at 0.65 bytes each. Calibrated from the official checkpoint profile.
13.7 GB
KV cache
Concurrency 1, context 4,096, 12 dense layers @ 4,096 tokens + 12 sliding-window layers @ 128 tokens, 8 KV heads, BF16 cache storage.
0.1 GB
Runtime / safety overhead
Conservative buffer for allocator fragmentation, kernels, and runtime scratch space.
1.5 GB
Weights = parameter count × bytes per parameter.
KV cache grows with context length, KV-bearing layers, concurrent requests, and the selected KV cache dtype.
Hybrid Qwen3.5 layers also keep a static linear-attention state, and runtime presets can inflate the required card VRAM beyond the core estimate.
Show the substituted formulas
Weights
parameter count × bytes per weight
21B × 0.65 = 13.7 GB
OpenAI's GPT-OSS model card lists a 12.8 GiB checkpoint for gpt-oss-20b. The estimator uses that published mixed MXFP4 + BF16 resident checkpoint size directly.
KV cache
batch × effective KV tokens across attention layers × 2 × KV heads × head dim × bytes per KV element
1 × 50,688 × 2 × 8 × 64 × 2 = 0.1 GB
Dense attention layers keep the full 4,096-token cache, while sliding-window layers only keep 128 tokens. BF16 controls the bytes per stored KV element.
Overhead
max(1.5 GB, 10% of weights + KV cache + linear state)
max(1.5 GB, 10% of 13.8 GB) = 1.5 GB
This leaves room for runtime buffers instead of claiming an unrealistically exact fit.
Tips
What to change next
Keep some spare VRAM on RTX 4090 24GB for runtime overhead instead of targeting a zero-margin fit.
Model
Selected model
GPT-OSS 20B
21B total • 3.6B active • 128,000 context • 8 KV heads
Architecture
Total params
Active params
Layers
Hidden size
Attention heads
KV heads
KV-bearing layers
Context length
Modality
License
Assumptions
The calculator works in raw bytes, displays decimal GB, and keeps both the core tensor footprint and the runtime-adjusted card requirement explicit instead of pretending every engine uses the full card the same way.