Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 26·5 min read
CyBOKClaw: Human-in-the-Loop CyBOK Mapping for Cybersecurity Curriculum
TL;DR
Cybersecurity education often lacks precise alignment with established knowledge frameworks, leading to confusion. CyBOKClaw is a human-in-the-loop retrieval framework that maps cybersecurity keywords to the Cyber Security Body of Knowledge (CyBOK).
✦ Why It Matters
Engineers can leverage CyBOKClaw to enhance the alignment of cybersecurity curricula with established knowledge frameworks.
Key Takeaways
- Utilize CyBOKClaw for curriculum development in cybersecurity.
- Implement human-in-the-loop approaches for better mapping accuracy.
- Focus on expert-guided evaluations for practical utility assessments.
- Adopt domain-sensitive ranking rules to improve retrieval outcomes.
Full Summary
Cybersecurity education faces challenges due to the broad and ambiguous nature of keywords used in the field, which often do not align well with the Cyber Security Body of Knowledge (CyBOK). CyBOKClaw was developed as an interpretable retrieval framework that generates top-k candidate mappings for expert review, rather than relying on strict exact classification.
It employs techniques such as query normalization, curated term expansion, and domain-sensitive ranking rules to enhance mapping accuracy. Evaluation metrics included Exact Match at top-5 (EXA-5) and Exact or Closest Acceptable Match at top-5 (ECA-5), with CyBOKClaw achieving 64.73% EXA-5 and 91.88% ECA-5 on the development dataset.
On the validation dataset, it improved to 81.19% EXA-5 and 98.00% ECA-5. These results indicate that expert-guided evaluations provide a more meaningful assessment of mapping utility than traditional exact matching methods.
CyBOKClaw serves as an effective tool for supporting curriculum development in cybersecurity education.