Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
A Geometric Account of Activation Steering through Angle-Norm Decomposition
TL;DR
Language models often struggle with controlling behavior through traditional methods. This study introduces a geometric approach called Angle-Norm Decomposition to better understand activation steering.
✦ Why It Matters
Engineers can enhance language model control by using distinct angular and radial parameters for activation steering.
Key Takeaways
- Explore geometric approaches for controlling language model behavior.
- Utilize distinct angular and radial components in activation steering.
- Recognize the importance of hidden-state norm for stability.
- Test different steering methods to understand their unique behaviors.
How It Works
Activation steering modifies a language model's output by adjusting its internal representations in a geometric space. The angular component aligns the model's state with specific concept directions, while the radial component adjusts the magnitude of the state.
This separation allows for more precise control over the model's behavior, enhancing the effectiveness of interventions.