EmbeddingGemma 2: an open, lightweight multimodal embedding model
deepmind.google·6d ago
AI Engineering
Hacker News·28d ago·5 min read
Hacker News·28d ago·5 min readWhen LLM judges agree, should we believe them?
TL;DR
A new method for aggregating outputs from multiple LLM judges improves evaluation accuracy by accounting for judge dependencies. This approach outperformed traditional voting methods by 9% to 14% across various tasks.
✦ Why It Matters
Evaluate your LLM judge panels for independence to enhance accuracy in assessments.
Key Takeaways
- Assess the judge panel for redundancy, not just individual performance.
- Model diversity should change error patterns for effective evaluation.
- Analyze agreement structures to uncover shared biases.
- Report confidence levels considering judge dependencies.
How It Works
The method models judges as a network, using Ising models to capture pairwise dependencies. It learns judge reliability and similarity from outputs without needing human reference labels, adjusting scores based on correlation to ensure diverse opinions are represented.