Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 21·6 min read
Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries
TL;DR
When AI models are unlearned (trained to forget specific information), researchers observed that answers appeared forgotten but internal reasoning traces still contained the forgotten content, raising questions about whether unlearning actually worked. The authors audited this claim on DeepSeek-R1-Distill-Qwen-7B using LoRA (a parameter-efficient fine-tuning method) to memorize fictional authors, then applied NPO unlearning while monitoring a six-token canary head (a probe detecting specific learned patterns).
✦ Why It Matters
Engineers evaluating unlearning in reasoning models need better auditing methods to distinguish genuine forgetting from trace-level artifacts before deploying safety-critical systems.
Key Takeaways
- Audit unlearning claims by testing whether reasoning traces genuinely leak memorized content or reflect other patterns.
- Use canary heads—small probes—to detect information leakage in model activations during unlearning evaluation.
- Swap reasoning traces between model states to isolate whether gaps indicate weight-level memorization or trace-level artifacts.
- Question standard interpretations of unlearning gaps; verify with targeted experiments before concluding weights retain forgotten data.
Full Summary
Machine unlearning aims to remove specific information from trained models, but evaluations sometimes show a puzzling pattern: final answers appear unlearned while the model's internal reasoning trace (the step-by-step thinking) still references forgotten content. This gap is often interpreted as evidence that weights retain the information.
Researchers conducted an audit on DeepSeek-R1-Distill-Qwen-7B, a reasoning model, using LoRA (Low-Rank Adaptation, a parameter-efficient fine-tuning method) to memorize fictional authors, then applied NPO unlearning to remove them. They used a six-token canary head—a small probe designed to detect whether specific information leaks through model activations.
By swapping reasoning traces between different model states, they tested whether the observed gap truly indicates memorization in weights or reflects a different phenomenon, challenging the standard interpretation of unlearning evaluation results.