Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 15·5 min read
Self-Referential Induction Increases Response Instability Relative to Unresolvable and Verifiable Questions in Large Language Models
TL;DR
Self-referential prompts in large language models lead to less consistent responses compared to unresolvable philosophical questions and verifiable queries. This study quantifies response instability, revealing self-referential questions produce the highest instability at 0.343.
✦ Why It Matters
Consider using more stable question types for consistent outputs in your applications today.
Key Takeaways
- Utilize unresolvable questions for more consistent model responses.
- Be cautious with self-referential prompts due to high instability.
- Measure response consistency when designing AI interactions.
- Explore the implications of subjective experience reports in applications.
How It Works
The study measures response instability by calculating the cosine similarity of sentence embeddings from repeated responses. This method quantifies how similar or different the model's outputs are when prompted with various types of questions.