Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 12·5 min read
Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints
TL;DR
An audit of 263 batch-normalized checkpoints reveals that unlearning metrics are influenced by the checkpoints' properties rather than the survival of removed data. This finding emphasizes the need for clear reporting of fitting conventions alongside published model numbers.
✦ Why It Matters
Ensure your model releases include fitting conventions to enhance transparency and trust in unlearning processes.
Key Takeaways
- Include fitting conventions with model releases for clarity.
- Understand that checkpoint properties influence unlearning metrics.
- Be cautious of unlearning claims without supporting data.
- Monitor the impact of retraining on model performance.
How It Works
The audit assesses unlearning by comparing published model metrics with those of retrained references. It specifically examines how batch-normalization statistics, which are not directly influenced by gradient updates, affect the model's performance after data removal.