Accelerating the search for antimicrobial peptides with Google Antigravity
deepmind.google·2h ago
AI Research
arXiv CS.AI·1d ago·5 min read
The Sharp Directions Are Against You: Curvature Analysis of Activation Steering
TL;DR
Activation steering is a lightweight method for controlling language model behavior by adding a vector to internal representations. However, its effectiveness is limited due to sensitivity in certain directions, leading to erratic behavior changes.
✦ Why It Matters
Evaluate your use of activation steering to ensure stability in model behavior during inference.
Key Takeaways
- Explore the use of activation steering for lightweight model control.
- Analyze the steering Hessian to identify sharp and flat directions.
- Remove sharp components from steering vectors to stabilize behavior.
- Consider geometric properties of vectors in model adjustments.
How It Works
Activation steering modifies a language model's internal representations by adding a vector, which can be analyzed using the steering Hessian. This matrix captures how sensitive the model's output is to changes in the steering vector, revealing which directions lead to erratic behavior.