We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·28d ago·5 min read
Native-speed vLLM transformers modeling backend
TL;DR
Hugging Face has integrated its transformers library as a modeling backend for vLLM, enabling seamless execution of transformer models with optimized inference techniques. This integration allows model authors to run their models without additional porting efforts.
✦ Why It Matters
Engineers can now run Hugging Face models in vLLM with minimal setup, enhancing their deployment options.
Key Takeaways
- Upgrade the vllm package to access new features.
- Use the --model-impl transformers flag for optimized performance.
- Explore the benefits of dynamic layer fusions in inference.
- Leverage the same model code for training and inference.
How It Works
The transformers modeling backend for vLLM uses torch.fx to analyze the model's computational graph, identifying patterns that can be optimized. It then rewrites operations to utilize vLLM's highly optimized kernels, allowing for efficient inference without requiring custom code.