We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
Hacker News·14d ago·5 min read
Ten ways a check passes while the thing it checks is broken
TL;DR
During an audit, a verification script passed despite serious underlying defects, highlighting the danger of relying on checks that have never failed. This incident underscores the need for rigorous testing practices.
✦ Why It Matters
Implement intentional failure tests to validate your verification checks today.
Key Takeaways
- Break the system intentionally to test checks.
- Use known positives to validate check outputs.
- Assess checks at the consequence layer, not just source.
- Design checks to capture real issues, not just pass.
How It Works
The audit identified that checks often passed due to their design, which allowed them to miss critical failures. For example, checks that never failed were shown to be unproven when subjected to intentional faults.
By employing a variety of testing methods, including checking the rendered output rather than just the source code, the audit revealed hidden issues that standard checks overlooked.