We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hugging Face Blog·Jun 29·5 min read
DiScoFormer: One transformer for density and score, across distributions
TL;DR
Estimating the underlying distribution of data points is challenging, especially in high dimensions. DiScoFormer is a new transformer model that simultaneously estimates both the density and score of a distribution in a single pass.
✦ Why It Matters
Engineers can use DiScoFormer to efficiently estimate data distributions without the need for retraining models.
Key Takeaways
- Utilize DiScoFormer for simultaneous density and score estimation.
- Leverage its adaptability for out-of-distribution data.
- Explore applications in generative modeling and Bayesian inference.
- Compare performance against traditional methods like KDE.
How It Works
DiScoFormer uses a transformer architecture with stacked layers and cross-attention to estimate density and score simultaneously. It leverages the mathematical relationship between score and density, allowing it to adapt to new data without retraining.
The model's shared backbone for both outputs ensures consistency, enabling it to learn multiple scales of influence across the data.