Trusted Data Apps for the Agentic Data Cloud, powered by Looker
cloud.google.com·7h ago
System Design
ByteByteGo·8d ago·5 min read
Last 3 days: AI Evals, October cohort
TL;DR
AI Evals in Practice is now open for enrollment, but only for the next three days. This program offers hands-on experience with AI evaluation techniques, making it a timely opportunity for interested participants.
✦ Why It Matters
Enroll in AI Evals in Practice today to enhance your AI evaluation skills before the deadline.
Key Takeaways
- Learn to design evaluations for AI quality and safety.
- Build and validate LLM-as-a-Judge systems effectively.
- Identify vulnerabilities in AI agents through red-teaming.
- Integrate evaluations into CI/CD pipelines for ongoing reliability.
Full Summary
AI Evals in Practice is a live course offered by ByteByteGo in collaboration with Salesforce's Manjeet Singh, focusing on creating robust evaluation systems for AI agents. Participants will gain skills in designing evaluations that assess quality, safety, reliability, cost, and latency.
The course includes practical training on building and validating LLM-as-a-Judge systems, which utilize large language models to assess AI performance. Additionally, attendees will learn to red-team agents to identify vulnerabilities such as prompt injection and jailbreaks.
The curriculum emphasizes the creation of meaningful evaluation datasets from both real and synthetic data, and covers evaluating tool use, retrieval-augmented generation (RAG), multi-step execution, and multi-agent handoffs. Finally, the course teaches how to integrate evaluations into continuous integration and continuous deployment (CI/CD) pipelines to detect regressions and drift, ensuring that failures are turned into permanent regression tests.