Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Jul 22·5 min read
Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles
TL;DR
Aggregating predictions from 15 large language models (LLMs) on binary questions demonstrated that learned aggregation methods significantly outperformed individual models. This suggests that LLM crowds can exhibit wisdom-of-crowds effects, but contamination in training data must be addressed for accurate assessments.
✦ Why It Matters
Ensure your model evaluations are free from training data contamination to achieve accurate performance assessments.
Key Takeaways
- Utilize learned aggregation methods for improved prediction accuracy.
- Evaluate models on clean datasets to avoid contamination effects.
- Consider logistic regression for effective model output combination.
- Explore the wisdom of crowds in your AI applications.
How It Works
The study employed learned aggregation techniques, specifically multilayer perceptrons and logistic regression, to combine outputs from multiple LLMs. These methods effectively learned to weigh the predictions based on model diversity, leading to superior performance compared to individual models.
⚠ The Catch
Training cutoff contamination was a significant confound, affecting the perceived accuracy of models and leading to misleading conclusions about their capabilities.