EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 27·5 min read
Benchmarking Opus 5 on SlopCodeBench
TL;DR
SlopCodeBench, a new coding benchmark, challenges models to evolve their codebase over time as new requirements emerge, achieving low pass rates of 11% and 17% for GPT-5.4 and Opus 4.6, respectively.
✦ Why It Matters
Explore SlopCodeBench to better evaluate your AI models' coding capabilities in evolving scenarios.
Key Takeaways
- Test AI models with SlopCodeBench for evolving codebase challenges.
- Monitor verbosity and complexity metrics in generated code.
- Consider human oversight for maintaining code quality.
- Explore the implications of checkpoint failures on future development.
How It Works
SlopCodeBench introduces a series of checkpoints that require AI models to build and adapt code incrementally, simulating real-world software engineering tasks. Each checkpoint builds on the previous one, and models must pass all tests to succeed, ensuring that defects from earlier stages affect later ones.