Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 26·5 min read
BODHI: Precise OS Kernel Specification Inference
TL;DR
Formal verification of operating system kernels requires precise specifications, which are difficult to create manually. BODHI is a new method that enhances large language models (LLMs) by using a structured translation guide to improve specification generation.
✦ Why It Matters
Engineers can leverage BODHI to improve the accuracy of OS kernel specifications using LLMs.
Key Takeaways
- Utilize BODHI to enhance OS kernel specification accuracy.
- Apply structured translation guides for better model performance.
- Explore different LLMs to find optimal configurations.
- Reduce errors in formal specifications with domain knowledge.
Full Summary
Operating system kernels need precise specifications to ensure their correct behavior, particularly for system calls. However, creating these specifications manually is challenging and requires extensive domain knowledge.
BODHI is a novel method that augments existing large language models (LLMs) with a structured C-to-Python translation guide, which organizes the specification process into distinct categories, such as pre-condition extraction and post-condition generation. Evaluated across nine models from six different providers, BODHI improved performance significantly, with accuracy gains ranging from 11% to 32%.
The best-performing configuration achieved a Pass@1 rate of 96.73%. BODHI effectively reduces both syntax and semantic errors, particularly benefiting models capable of following structured instructions.
This demonstrates that integrating domain knowledge can enhance the capabilities of general-purpose code generation tools for formal specification synthesis.