Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning
TL;DR
Post-training pruning of large language models (LLMs) often uses a small unlabelled calibration set, but its impact on accuracy may vary across different capabilities. This study evaluates the effects of calibration data from multiple sources on LLM performance in areas like General knowledge, Commonsense reasoning, Code generation, and Math.
✦ Why It Matters
Engineers should choose calibration data sources strategically to optimize LLM performance across specific capabilities.
Key Takeaways
- Evaluate calibration data sources for specific model capabilities.
- Consider distinct performance metrics for General, Commonsense, Code, and Math.
- Use multiple calibration sources to enhance model accuracy.
- Analyze the impact of calibration on post-pruning performance.
Full Summary
Large language models (LLMs) can be compressed through post-training pruning, which reduces their size while maintaining performance. Previous research suggested that the source of calibration data—used to fine-tune the model—had a modest effect on overall accuracy.
However, this study investigates whether this conclusion holds when examining specific capability dimensions: General knowledge, Commonsense reasoning, Code generation, and Math. By analyzing 15 different calibration sources, the researchers found that the choice of calibration data significantly impacts performance in these areas.
For instance, certain sources improved accuracy in Commonsense reasoning but not in Math. These findings suggest that engineers should carefully select calibration data based on the specific capabilities they wish to enhance in their models.