Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 28·5 min read
Probing for Knowledge Attribution in Large Language Models
TL;DR
Large language models (LLMs) often lack transparency regarding their knowledge sources, making it difficult to understand their reasoning. This study introduces a probing technique that identifies and attributes knowledge within LLMs, specifically using a method called 'knowledge probing.'
✦ Why It Matters
Engineers can use knowledge probing to enhance the interpretability and trustworthiness of their AI models.
Key Takeaways
- Use AttriWiki to generate labeled training data for LLMs.
- Train linear probes to identify knowledge sources in model outputs.
- Monitor attribution mismatches to reduce error rates significantly.
- Apply findings to improve the reliability of LLM applications.
How It Works
AttriWiki operates by prompting LLMs to recall specific entities from memory or context, creating a dataset of labeled examples. This data is then used to train linear probes that analyze hidden representations of the model's outputs, allowing for accurate classification of the knowledge source behind each response.