Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Towards Data Science·Sep 4·5 min read
Disaggregation Is a Thousand-GPU Problem
TL;DR
Disaggregation of prefill from decode in GPU systems is only beneficial under specific conditions. Below this threshold, chunked prefill remains the optimal approach.
✦ Why It Matters
Evaluate your GPU workload characteristics to determine if disaggregation is beneficial for your system today.
Key Takeaways
- Use chunked prefill to reduce scheduling interference on shared GPUs.
- Measure p95 time-per-output-token to assess performance issues.
- Avoid disaggregation unless you have thousands of GPUs and fast interconnects.
- Profile your workload to determine the prefill fraction of total time.
How It Works
Chunked prefill divides long prefill requests into smaller segments, allowing them to be processed alongside decode batches on the same GPU. This interleaving reduces resource contention and improves overall throughput without requiring separate GPU pools or complex network transfers.