Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
LLMs Can Better Capture Human Judgments--With the Right Prompts
TL;DR
Large language models (LLMs) struggle to accurately reflect human judgments due to limitations in capturing response distributions and stability. By employing specific prompting techniques, such as asking for standard deviations and ensuring clarity in scenarios, LLMs can better align with human responses.
✦ Why It Matters
Engineers can enhance LLM performance by using targeted prompting techniques to improve human-AI alignment.
Key Takeaways
- Use standard deviations in prompts to capture response variability.
- Ensure clarity in scenarios to improve model alignment.
- Evaluate human confusion ratings to refine prompts.
- Recognize LLMs' limitations in self-assessing error.
How It Works
The study employs prompting techniques that ask LLMs to report not just their answers but also the variability of those answers. By doing so, the models can better reflect the range of human responses, leading to improved alignment with actual human judgments.