Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Simon Willison's Blog·Jul 22·5 min read
Are AI labs pelicanmaxxing?
TL;DR
Dylan Castillo's investigation into AI models reveals no evidence that labs are specifically training them to draw pelicans on bicycles. Results show that performance on this task is not significantly better than for other animals or vehicles.
✦ Why It Matters
Consider testing your AI models against unconventional benchmarks to explore their creative limits.
Key Takeaways
- Explore unconventional benchmarks to assess AI creativity.
- Use diverse prompts to evaluate model performance.
- Analyze results across multiple AI models for comprehensive insights.
- Stay updated on AI research methodologies for better evaluations.
Full Summary
Dylan Castillo conducted a thorough analysis to determine if AI labs were intentionally enhancing their models to depict pelicans riding bicycles, a concept humorously suggested by Simon Willison. He tested 48 prompts involving eight animals and six vehicles across seven AI models, including GPT-5.6 Terra and Claude Sonnet 5.
Each prompt was evaluated three times, with additional assessments from GPT-5.6 Luna and Gemini 3.1 Flash-Lite. The findings indicated no significant improvement in the quality of pelican-bicycle images compared to other combinations.
Notably, GLM-5.2 showed a slight edge in this specific category, but the overall effect was minimal. This suggests that AI models do not exhibit specialized capabilities for this whimsical task, reinforcing the idea that their performance is consistent across various subjects and scenarios.