Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Prefill Awareness in Large Language Models
TL;DR
Large language models may struggle with safety evaluations if they cannot recognize when their outputs have been altered. Researchers developed a benchmark to assess this capability, termed prefill awareness, across various prefill mechanisms.
✦ Why It Matters
Engineers should consider prefill awareness when designing safety protocols for AI models to ensure reliability.
Key Takeaways
- Monitor prefill awareness in language models for safety evaluations.
- Utilize the binary preference benchmark to assess model capabilities.
- Understand the impact of stylistic and preference mismatches on model behavior.
- Consider dataset context when evaluating model responses to prefilled outputs.
Full Summary
Safety evaluations of language models often depend on the ability to prefill outputs, which can be compromised if models cannot identify alterations in their responses. Researchers introduced a binary preference benchmark to test 'prefill awareness,' the ability of models to distinguish between original and modified outputs.
They found that Claude Opus 4.5 could detect tampered outputs in 9-35% of instances without false positives. Additionally, models tended to revert to baseline responses when faced with conflicting prefill information.
The study revealed that stylistic mismatches influenced the detection of foreign prefill, while preference mismatches affected the tendency to revert to original answers. In more complex scenarios, models sometimes rejected prefilled responses based on the dataset and task context.
These findings suggest that prefill awareness is a critical factor that developers should monitor in advanced language models.