NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks
TL;DR
Large language models (LLMs) are being tested for cybersecurity tasks, but their readiness is unclear. Researchers created dual-mode benchmarks: white-box code vulnerability detection and black-box web application security testing across six frontier models and two specialized models.
✦ Why It Matters
Engineers should prioritize domain-specialized security models and structured methodology over scaling general LLMs for vulnerability detection.
Key Takeaways
- Frontier models produce 10-50% false positives; methodology beats scale for security tasks.
- Black-box detection achieves only 4-8% coverage; external tools improve to 10-19% maximum.
- Domain-specialized agents with structured penetration testing exceed 50% per-family detection rates.
- Build training data with structured traces, request/response sequences, and multi-step attack chains.
Full Summary
Current frontier LLMs (GPT-5.4, Claude Opus, Gemini Pro) lack proven effectiveness for cybersecurity tasks despite their general capabilities. Researchers evaluated six frontier models and two domain-specialized models using a dual-mode benchmark: VulnLLM-R for white-box function-level vulnerability detection across C, Java, and Python; and black-box testing on five production-style applications containing 118 ground-truth vulnerabilities spanning 20+ CWE families (Common Weakness Enumeration categories).
Frontier models systematically over-predicted vulnerabilities, producing 10-50% false positive rates in white-box detection and achieving only 4-8% ground-truth coverage in black-box testing, improving marginally to 10-19% with external security tools like Playwright and Burp Suite. Domain-specialized agents using structured penetration-testing methodology raised per-family detection above 50%, while a specialized defense model achieved 0.904 precision and 9.7% false positive rate on a single GPU.
The fundamental bottleneck is training data: absence of structured security testing traces, end-to-end request/response sequences, failure-heavy data, and multi-step attack chains. Researchers propose self-play security testing as a data generation strategy and argue for vertical foundation models purpose-built for cybersecurity rather than relying on general-purpose LLMs.