We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 2·5 min read
The gauge broke: devs felt 20% faster with AI, measured 19% slower
TL;DR
Experienced developers reported feeling 20% faster with AI tools, but a study found they were actually 19% slower. A controlled trial by METR involved 16 developers working on familiar codebases using current AI tools.
✦ Why It Matters
Engineers should critically evaluate AI tools' impact on productivity, especially for experienced developers in established codebases.
Key Takeaways
- Assess AI tools' impact on productivity critically.
- Recognize the difference between perceived and actual speed.
- Consider team experience when implementing AI solutions.
- Evaluate AI effectiveness in both new and existing projects.
Full Summary
For two years, there was a growing concern that the perceived speed boost from AI tools did not match actual productivity gains. A controlled trial conducted by METR involved 16 experienced open-source developers working on familiar codebases with advanced AI tools.
Developers expected these tools to enhance their speed, and they reported a 20% increase in perceived efficiency. However, the measured results showed they were approximately 19% slower in completing tasks.
This significant gap of nearly 40 percentage points between self-reported speed and actual performance suggests that experienced developers may overestimate the benefits of AI. The study indicates that while AI tools can be beneficial for junior developers and new projects, they may hinder productivity for seasoned professionals working with existing code.
These findings urge engineering leaders to reconsider how they assess team velocity.