How to Choose the Right GPU for vLLM Inference

Large language model GPU selection requires understanding two processing phases: prefill (memory-bandwidth bound, affects time-to-first-token) and decode (compute-bound, determines generation speed). VRAM capacity sets absolute limits, with a 70B model requiring 140GB in FP16 just for weights, making quantization essential. The KV cache grows dynamically based on context length and concurrent users, consuming additional memory during runtime. Quantization is the primary optimization tool, with FP8 offering excellent speed-quality balance, while tensor parallelism enables multi-GPU scaling when models exceed single-GPU capacity.