Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Simon Willison's Blog·Jul 31·5 min read
smevals - a small eval suite for evaluating models, prompts, and harnesses
TL;DR
smevals is a new evaluation suite designed for assessing AI models, prompts, and harnesses, developed in collaboration with Jesse Vincent's Prime Radiant lab. It allows users to create and run customized evaluation suites to better understand model capabilities.
✦ Why It Matters
Start using smevals today to create tailored evaluation suites for your AI models.
Key Takeaways
- Run 'uvx smevals docs' to access the tool's documentation.
- Create evals using YAML files to define tasks and challenges.
- Grade model runs with 'uvx smevals grade' to assess performance.
- Serve results locally or generate static HTML reports for sharing.
How It Works
smevals operates by allowing users to define evals as directories containing YAML files that specify tasks and configurations. Users can run these evals against selected models, recording the outcomes of each task execution.
The grading process involves running a series of checks on the outputs, which can be simple validations or more complex evaluations using additional models.