Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·16d ago·5 min read
Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities
TL;DR
Autonomous vehicles face significant security threats, necessitating structured threat information generation. Open-weight large language models (LLMs) were evaluated for their effectiveness in this task.
✦ Why It Matters
Engineers can implement open-weight LLMs to automate the generation of threat assessments for autonomous vehicle security.
Key Takeaways
- Utilize CAV-STIXGen for structured threat information generation.
- Evaluate open-weight LLMs for vulnerability mapping in your systems.
- Focus on improving SRO and MITRE ATT&CK mappings for better threat analysis.
- Implement AI-assisted tools to enhance transportation security measures.
Full Summary
Connected and Autonomous Vehicles (CAVs) are increasingly reliant on complex software and hardware systems, making them susceptible to various vulnerabilities. These vulnerabilities are often recorded in the Common Vulnerabilities and Exposures (CVE) database, but security professionals need structured information to effectively address them.
To facilitate this, a dataset called CAV-STIXGen was created, mapping CAV vulnerabilities to Structured Threat Information Expression (STIX) formats. Eleven open-weight LLMs were tested, with single models achieving F1 scores of 0.94 for Structured Domain Objects (SDO) and 0.99 for Common Weakness Enumeration (CWE) mappings.
However, mapping to STIX Relationship Objects (SRO) and complete MITRE ATT&CK techniques proved more challenging. In a multi-agent setup, models like Gemma-4-31B showed promising results, achieving an F1 score of 0.91 for SDOs.
This research highlights how AI can automate the translation of vulnerabilities into structured threat information, enhancing defense strategies in transportation security.