We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·20d ago·5 min read
J-space comparisons across open models
TL;DR
J-space analysis reveals how nudges in a model's middle layers influence output over time. By measuring this influence across open models, researchers discovered the extent and timing of these effects.
✦ Why It Matters
Engineers can use these findings to refine model architectures for better output control and performance.
Key Takeaways
- Focus on early layers for sustained output influence.
- Utilize CKA to compare layer influence across models.
- Measure participation ratios to assess effective directions.
- Re-evaluate training strategies based on temporal influence findings.
How It Works
The study measures how nudging a model's residual stream at different layers affects the output. By creating influence matrices, researchers can quantify how long a nudge impacts future tokens.
CKA is used to compare the shape of influence dictionaries across layers, while PR indicates the number of effective directions utilized by each layer.