Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 27·6 min read
When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning
TL;DR
In-context learning (ICL)—where language models learn from input-output examples in their prompt—assumes correct demonstrations improve accuracy. Researchers discovered this assumption fails: some correct examples actually hurt performance.
✦ Why It Matters
Select in-context demonstrations based on relevance and distribution fit, not just correctness, to avoid degrading model accuracy.
Key Takeaways
- Evaluate how demonstrations influence model inference.
- Consider perturbations to test exemplar utility.
- Focus on the complexity of tasks when using ICL.
- Be cautious with smaller models and high perturbation ratios.
How It Works
The study introduces task-preserving perturbations, which modify only the input of an exemplar while maintaining its correctness. This approach allows researchers to analyze how changes in input affect the model's inference process, revealing that even correct examples can mislead the model.