NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 27·6 min read
Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal
TL;DR
Large reasoning models (LRMs) that generate chain-of-thought (CoT) traces—intermediate reasoning steps before final answers—complicate traditional refusal mechanisms designed for simpler language models. Researchers tested activation steering, a technique that manipulates internal model states to control behavior, on DeepSeek-R1-Distill-LLaMA-8B.
✦ Why It Matters
Engineers building safety controls for reasoning models must account for chain-of-thought traces, not just final outputs, when implementing refusal mechanisms.
Key Takeaways
- Explore the impact of chain-of-thought on model refusals.
- Implement two-stage interventions to improve compliance rates.
- Analyze the role of activation states in refusal behaviors.
- Consider vulnerabilities in chain-of-thought processes for security.
How It Works
LRMs utilize chain-of-thought (CoT) to generate outputs, which complicates refusal mechanisms. The study shows that when the CoT is fixed, refusal reversal is limited.
However, regenerating the CoT during interventions significantly enhances compliance, indicating that the CoT can independently carry compliance signals.