Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·Aug 10·5 min read
Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons
TL;DR
A study reveals that short-budget AutoML comparisons can be misleading due to protocol flaws. Orcetra's apparent success over FLAML and AutoGluon was largely due to unregulated budget enforcement and biased test set selection.
✦ Why It Matters
Ensure your AutoML comparisons enforce strict budget and evaluation protocols to avoid misleading results.
Key Takeaways
- Always enforce budget limits during AutoML evaluations.
- Use validation splits instead of test sets for candidate selection.
- Be cautious of inflated performance metrics from repeated test evaluations.
- Follow a checklist for fair AutoML comparisons.
How It Works
Orcetra's evaluation process involved scoring each candidate multiple times on the test set, which led to inflated performance metrics. In contrast, FLAML and AutoGluon only evaluated candidates once, providing a more accurate representation of their capabilities.
⚠ The Catch
The lack of enforced budget during candidate evaluations allowed Orcetra to exceed its nominal time limit, skewing the results in its favor.