NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
EDGE-OPD: Internalizing Privileged Context with Evidence Guided On-Policy Distillation
TL;DR
Training large language models with privileged information (extra context unavailable at inference) causes unintended side effects like altered reasoning and degraded general capabilities. EDGE-OPD (Evidence Guided On-Policy Distillation) addresses this by using guided rollouts to ensure target behavior appears in training data and applying an evidence mask to update only tokens supported by privileged context.
✦ Why It Matters
Engineers can use EDGE-OPD to train models with privileged information without degrading general performance or introducing unintended behavioral changes.
Key Takeaways
- Use guided rollouts to ensure target behaviors actually appear in training data before distillation.
- Apply evidence masks to update only tokens where privileged context genuinely supports the prediction.
- Test for unintended side effects like reasoning changes and response length shifts when using privileged information.
- Validate that general-purpose capabilities remain intact after training with privileged context.
Full Summary
On-Policy Distillation (OPD) is a training approach for large language models that improves performance without causing distribution drift—unwanted shifts in model behavior on unrelated tasks. On-Policy Self-Distillation (OPSD), a variant using a single model as both teacher and student, can leverage privileged context (persona, private facts, worked solutions) during training that won't exist at inference time.
However, this privileged information often causes unintended side effects: it modifies reasoning patterns, degrades general capabilities, and changes response characteristics like length or style. EDGE-OPD introduces two key modifications: guided rollouts that inject target behavior into training data so the model actually sees examples of the desired behavior, and an evidence mask that updates the student model only at token positions where privileged context genuinely supports the generated token.
Experiments on rare-token and identity tasks show OPSD and its variants completely fail to learn target identities, while EDGE-OPD succeeds. Ablation studies reveal persona signals concentrate in positive-evidence regions, providing insights into efficient knowledge transfer and general capability preservation.