Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models
TL;DR
Existing knowledge-editing methods for autoregressive language models struggle with masked diffusion language models (MDLMs) due to high resource demands. TimeROME-DLM is introduced as a training-free, gradient-free framework that enables efficient knowledge editing in MDLMs using a causal-tracing protocol and low-rank memory updates.
✦ Why It Matters
Engineers can implement TimeROME-DLM for efficient knowledge editing in MDLMs without incurring high computational costs.
Key Takeaways
- Utilize TimeROME-DLM for efficient knowledge editing in MDLMs.
- Implement causal-tracing protocols to enhance model predictions.
- Leverage low-rank memory updates to minimize resource usage.
- Achieve significant speed improvements in processing time.
Full Summary
Masked diffusion language models (MDLMs) like LLaDA are becoming competitive with autoregressive (AR) models, but current knowledge-editing techniques are not effective for them. TimeROME-DLM is a novel framework that allows for inference-time knowledge editing without requiring additional training or gradient updates.
It employs a Temporal Indirect Effect (TIE) causal-tracing protocol to identify key factors influencing predictions and utilizes a low-rank residual edit memory to apply updates efficiently. In experiments, TimeROME-DLM reduced forget-set log-probability by approximately 83 nats while keeping the performance on retained facts stable.
It also demonstrated a four- to fourteen-fold speedup in processing time without increasing VRAM usage. This approach scales well, handling up to 400 facts with minimal computational cost, effectively bridging the gap between AR and MDLMs.
These advancements can significantly enhance the efficiency of knowledge management in language models.