EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·Jul 22·5 min read
Are AI Labs Pelicanmaxxing?
✦ Why It Matters
Consider testing your AI models against unconventional benchmarks to uncover hidden biases.
Key Takeaways
- Explore diverse prompts for evaluating AI models.
- Avoid relying solely on popular benchmarks for performance assessment.
- Analyze model outputs quantitatively for better insights.
- Consider the difficulty of tasks when interpreting results.
Full Summary
Simon Willison's humorous benchmark of generating an SVG of a pelican on a bicycle has gained traction in AI discussions. To investigate if AI labs are optimizing for this benchmark, an experiment was conducted generating 1,008 SVGs using seven models, including GPT-5.6 Terra and Claude Sonnet 5.
Each image was scored for animal and vehicle quality, revealing that pelicans ranked sixth out of eight animals and bicycles were second to last among vehicles. A regression analysis confirmed that no lab showed a significant boost in performance for pelicans or bicycles, suggesting that AI models are not specifically trained to excel at this benchmark.
The findings imply that while the pelican-on-a-bicycle prompt is popular, it may not reflect the true capabilities of AI models.