Will It Fit?
Estimate text inference VRAM across Transformers and vLLM with a compact, explainable breakdown.
Model pages and release notes are included here too.
Qwen 3.5 35B A3B · vLLM · Official BF16 checkpoint
Required GPU VRAM (0.9 budget)
88.1 GB
Core estimate: 79.3 GB. Against RTX 4090 24GB, this leaves 62.3 GB of deficit.
At 4,096 tokens, the estimated max concurrency is 0 concurrent requests.
Quick read
Selected GPU
Runtime
Load dtype
GPU utilization
GPU count
KV cache dtype
Context length
Current concurrency
Max concurrency @ context
Class
Bandwidth
Nominal VRAM
Core estimate
Required GPU VRAM (0.9 budget)
Deficit
GPU compare
What fits this setup
Breakdown
Where the memory goes
Weights
35B resident parameters at 2.05 bytes each. Calibrated from the official checkpoint profile.
71.9 GB
KV cache
Concurrency 1, context 4,096, 10 KV-bearing layers, 2 KV heads, BF16 cache storage.
0.1 GB
Linear attention state
Concurrency 1, 30 linear-attention layers, static recurrent state, and short-convolution buffers. This term stays flat as context grows.
0.1 GB
Runtime / safety overhead
Conservative buffer for allocator fragmentation, kernels, and runtime scratch space.
7.2 GB
Weights = parameter count × bytes per parameter.
KV cache grows with context length, KV-bearing layers, concurrent requests, and the selected KV cache dtype.
Show the substituted formulas
Weights
parameter count × bytes per weight
35B × 2.05 = 71.9 GB
The official Qwen3.5-35B-A3B checkpoint totals about 71.90 GB on Hugging Face, and Qwen documents Transformers, vLLM, and related serving stacks for the release.
KV cache
batch × effective KV tokens across attention layers × 2 × KV heads × head dim × bytes per KV element
1 × 40,960 × 2 × 2 × 256 × 2 = 0.1 GB
Only the attention-bearing layers contribute KV cache in this hybrid stack, and BF16 controls the bytes per stored KV element.
Linear state
batch × linear layers × (recurrent state + short-conv buffers) × state bytes
1 × 30 × 557,056 × 4.00 = 0.1 GB
Hybrid Qwen3.5 layers keep a static recurrent state plus q/k/v short-convolution buffers. The published configs keep that state in float32, so it is modeled separately from the BF16 weight dtype.
Overhead
max(1.5 GB, 10% of weights + KV cache + linear state)
max(1.5 GB, 10% of 72.1 GB) = 7.2 GB
This leaves room for runtime buffers instead of claiming an unrealistically exact fit.
Model
Selected model
Qwen 3.5 35B A3B
35B total • 3B active • 262,144 context • 2 KV heads
Total params
Active params
Layers
Hidden size
Attention heads
KV heads
KV-bearing layers