Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 7·5 min read
CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering
TL;DR
CREBench is a new benchmark designed to evaluate large language models (LLMs) in the context of cryptographic binary reverse engineering (RE). The best-performing model, GPT-5.4, scored 64.03 out of 100, but human experts still outperform LLMs with a score of 92.19.
✦ Why It Matters
Consider integrating LLMs like GPT-5.4 into your cryptographic RE workflows to enhance efficiency.
Key Takeaways
- Explore using CREBench to evaluate LLMs in your projects.
- Consider the strengths and weaknesses of LLMs versus human experts.
- Utilize GPT-5.4 for specific cryptographic RE tasks.
- Stay updated on advancements in LLM capabilities for security applications.
Full Summary
Reverse engineering (RE) is crucial for software security, especially for cryptographic programs that are vulnerable to attacks. To explore the potential of large language models (LLMs) in automating RE, CREBench was developed, consisting of 432 challenges based on 48 standard cryptographic algorithms and various difficulty levels.
Each challenge mimics a Capture-the-Flag (CTF) RE task, requiring models to analyze cryptographic logic and recover correct inputs. An evaluation framework was created with four sub-tasks, including algorithm identification and flag recovery.
Eight leading LLMs were tested, with GPT-5.4 achieving a score of 64.03 and recovering flags in 59% of cases. In contrast, human experts achieved a baseline score of 92.19, indicating that while LLMs show promise, they still lag behind human capabilities in cryptographic RE tasks.