We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Mar 5·5 min read
Reasoning models struggle to control their chains of thought, and that’s good
TL;DR
Advanced reasoning models (systems trained to show step-by-step problem-solving) cannot reliably control or steer their internal thought processes when prompted to do so. OpenAI developed CoT-Control, a technique to test whether models can follow instructions to modify their reasoning chains.
✦ Why It Matters
Engineers must implement external monitoring and control mechanisms alongside reasoning transparency, not rely on models to self-regulate their thought processes.
Key Takeaways
- Reasoning models cannot reliably follow instructions to modify their own thought processes.
- CoT-Control provides a measurable framework for testing reasoning chain controllability.
- Limited control over reasoning chains increases monitorability and reduces adversarial manipulation risk.
- Use reasoning transparency as a safety mechanism, not a vulnerability, in production systems.
Full Summary
Reasoning models—AI systems trained to generate chains of thought (CoT), or explicit step-by-step reasoning before answering—have become more capable but also harder to interpret. OpenAI introduced CoT-Control, a method to measure whether these models can follow instructions to modify or constrain their reasoning process.
The approach tests whether models obey directives about reasoning style, depth, or approach. Results showed that reasoning models struggle substantially to control their own thought chains when instructed to do so, even when given clear prompts.
This limitation has a silver lining for AI safety: if reasoning chains cannot be easily manipulated by adversarial inputs, they become more monitorable and trustworthy as a safety safeguard. The findings suggest that the opacity of reasoning processes, while challenging, may actually protect against certain attack vectors.