Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·10d ago·5 min read
Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?
TL;DR
A benchmark study reveals that off-the-shelf small language models (SLMs) fail to meet eligibility thresholds for microtasks in agent harnesses, with no configurations passing the tests. The findings suggest that SLMs should only be used when a baseline model fails to meet confidence thresholds.
✦ Why It Matters
Evaluate your current model's eligibility against established thresholds before deploying small language models for microtasks.
Key Takeaways
- Test small language models against defined eligibility thresholds.
- Use a baseline model that meets confidence intervals before SLMs.
- Quantization can negatively impact model performance.
- Consider model size when evaluating eligibility for tasks.
How It Works
The study establishes a benchmark for microtasks with fixed prompts and automatic metrics, requiring models to meet a confidence threshold to be considered eligible. It employs a logprob decision-threshold diagnostic to categorize failures into capability deficits and those addressable by adjusting decoding thresholds.