We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Reddit r/LocalLLaMA·May 20·5 min read
Move to backend sampling for MTP draft path by gaugarg-nv · Pull Request #23287 · ggml-org/llama.cpp
TL;DR
llama.cpp's MTP (multi-token prediction) draft path previously sampled tokens on CPU, creating a bottleneck where computation happened on GPU but token selection happened on slower CPU. A pull request moved token sampling to the GPU backend, keeping the entire operation on accelerated hardware.
✦ Why It Matters
Engineers optimizing local LLM inference can expect faster token generation and reduced latency by keeping sampling operations on GPU rather than bouncing between devices.
Key Takeaways
- Move sampling operations from CPU to GPU to eliminate synchronization overhead in draft paths.
- Backend sampling maintains numerical equivalence while reducing latency in speculative decoding.
- GPU-resident sampling prevents expensive CPU-GPU round-trip transfers during token generation.
- Apply this optimization pattern to other inference bottlenecks involving CPU-GPU data movement.
Full Summary
llama.cpp is a C++ inference engine for large language models that supports speculative decoding—a technique where a smaller model generates multiple candidate tokens (draft tokens) quickly, then a larger model validates them in parallel. MTP (Multi-Token Prediction) is a variant that predicts several tokens simultaneously.
Previously, the sampling step (randomly selecting tokens based on probability distributions) occurred on CPU after GPU computation, forcing expensive synchronization between devices. The pull request #23287 refactored the sampling logic to execute entirely on GPU backend, keeping computation on the accelerator.
This eliminates round-trip data transfers and CPU-GPU stalls. The change maintains numerical equivalence with prior behavior while reducing latency in the draft path—the critical path for speculative decoding performance.