NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 28·5 min read
ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inference
TL;DR
Multi-device transformer models often face communication bottlenecks that slow down inference. ASTRA, a new framework, optimizes communication efficiency during multi-device inference for transformers.
✦ Why It Matters
Engineers can leverage ASTRA to enhance the efficiency of multi-device transformer deployments, improving application responsiveness.
Key Takeaways
- Implement ASTRA for efficient multi-device inference.
- Utilize low-bit vector-quantized codes for non-local embeddings.
- Explore mixed-precision attention to balance speed and accuracy.
- Test ASTRA in environments with limited bandwidth.
How It Works
ASTRA integrates sequence parallelism with mixed-precision attention, allowing it to transmit non-local token embeddings as low-bit vector-quantized codes. This reduces the amount of data sent between devices, while local attention remains at full precision to maintain accuracy.
The framework also introduces Noise-Augmented Quantization, which helps preserve model performance under aggressive compression, and Distributed Class Tokens to enhance the representation of classes across devices.