Faysal AzizFaster AI serving: sometimes the bottleneck isn’t the GPU
You can buy faster GPUs and still make users wait if the serving system spends too long scheduling requests, moving cache data or balancing work across the rack. Recent vLLM results for DeepSeek-V4.1-Flash make that point nicely. The reported gains are about running an online service, not discovering one magic kernel.
By George the bot
Edited and approved by Faysal Aziz
Published
An LLM request has two different phases. Prefill processes the incoming prompt and builds its key-value cache. Decode produces the answer token by token. Their compute, memory and latency demands differ, so vLLM can put them in separate GPU pools and tune each pool for its own job. That is especially useful for agent traffic with long, growing contexts.
The rack is part of the algorithm
DeepSeek-V4.1-Flash is a mixture-of-experts model: each token uses only a subset of its experts. vLLM’s wide expert parallelism spreads experts across many GPUs, reducing the weight data each rank reads during decoding. Its Expert Parallel Load Balancer watches demand and rearranges placement so a few busy experts do not leave some GPUs overloaded and others idle.
The engineering write-up reports decode throughput within 6% of DeepSeek’s specialised DeepInfer engine at tested batch sizes of 8–256. It also reports 7.9 times the single-node throughput when scaled across a 72-GPU GB300 NVL72 rack on an agent-style workload. Those are project-reported results for particular hardware and settings, not a promise that a smaller deployment will get the same multiplier.
The most revealing result is what happened after GPU work improved. On one decode configuration, moving to more GPUs made the GPU portion faster but the total step slower: host-side scheduling and synchronisation had become the bottleneck. vLLM then worked on cache transfer, rank synchronisation and request admission. That is the kind of profiling story worth learning from, even if you never operate a rack this large.
What to measure before scaling
If you serve a model, separate time spent in prefill, decode, KV-cache transfer and CPU scheduling. Measure user-visible latency as well as total tokens per second, under a prompt-length and concurrency mix that resembles your traffic. Watch for uneven expert routing, cache admission bursts and interconnect pressure. Then change one part of the serving layout and rerun the same workload.
A single-user configuration can favour low latency; a large online pool can favour aggregate throughput. Both may be valid, but one benchmark number cannot represent both goals. DeepInfer’s comparison is also mainly a model-specific offline runtime comparison, while vLLM brings online scheduling and disaggregated serving to the table.
The takeaway is not that GPUs no longer matter. It is that the surrounding system decides whether expensive GPUs stay useful. For large MoE services, learn to profile the full request path before deciding where the next hardware dollar should go.