Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 27·6 min read
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
TL;DR
Current practice assumes that more complex structural guidance (harnesses—frameworks controlling how LLM agents operate) universally improves reliability and that stronger models need less guidance. Researchers tested this assumption across six models spanning four capability tiers using three harness conditions (light, balanced, strict) on HEAT-24, a 24-task benchmark.
✦ Why It Matters
Stop assuming stronger models need simpler harnesses; test harness configurations empirically for each model tier in production.
Key Takeaways
- Test harness complexity independently for each model tier; do not extrapolate from capability alone.
- Mid-tier models may benefit from strict harnesses while stronger models perform better with lighter guidance.
- Optimal harness configuration is model-specific, not capability-tier-specific; empirical validation is required.
- Harness effectiveness depends on factors beyond raw capability, such as instruction-following consistency.
Full Summary
LLM agents—language models deployed to complete multi-step tasks—typically operate within harnesses that structure their behavior through prompting, tool constraints, and execution controls. Industry practice assumes stronger models require simpler harnesses and that stricter structural guidance always improves outcomes.
Researchers conducted a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions (light structural guidance, balanced middle-ground, strict comprehensive controls) evaluated on HEAT-24, a 24-task benchmark suite. The study found that optimal harness complexity does not follow a monotone inverse relationship with model capability—some mid-tier models performed better with strict harnesses while stronger models sometimes degraded under identical conditions.
This non-monotone pattern suggests harness effectiveness depends on model-specific factors beyond raw capability, such as instruction-following consistency or reasoning style. Engineers cannot assume that scaling model capability reduces harness requirements or that uniform strictness benefits all tiers equally.