NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
TL;DR
Machine unlearning in Mixture-of-Experts (MoE) language models faces challenges due to routing mismatches between forget and retain data. TRACE, or Targeted Routing-Aware Calibration of Experts, was developed to address this by reweighting retain losses based on expert activation frequencies.
✦ Why It Matters
Engineers can implement TRACE to improve machine unlearning processes in MoE language models, enhancing model performance and compliance.
Key Takeaways
- Utilize TRACE for better expert calibration in MoE models.
- Address activation mismatches during machine unlearning.
- Evaluate performance improvements on datasets like WMDP and MUSE-BOOKS.
- Aim for a balance between forgetting and retaining critical data.
How It Works
TRACE identifies forget-critical experts by analyzing offline activation statistics. It then recalibrates the regularization of retained data by reweighting losses, ensuring that the activation frequencies of experts for retained data match those for forgotten data.
This targeted approach helps maintain the performance of critical experts during the unlearning process.