NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming
TL;DR
Large Language Models (LLMs) have not been thoroughly evaluated for their educational support in competitive programming. UOJ-Bench was developed to assess LLMs on tasks like code generation, hacking, and repair using real-world submissions.
✦ Why It Matters
Engineers can leverage UOJ-Bench to enhance LLMs for educational applications in competitive programming.
Key Takeaways
- Explore UOJ-Bench for evaluating LLMs in programming tasks.
- Consider test-time scaling to improve error detection rates.
- Utilize insights from LLMs to enhance educational tools.
- Investigate the balance between computational cost and model performance.
Full Summary
Competitive programming often relies on human judgment to identify errors in code submissions, but the potential of Large Language Models (LLMs) in this context remains underexplored. UOJ-Bench is a new benchmark designed to evaluate LLMs on three tasks: code generation (creating new code), code hacking (finding vulnerabilities), and code repair (fixing errors), using data from the Universal Online Judge (UOJ).
The evaluation process leverages UOJ's existing judging infrastructure to assess model performance. Findings reveal that under one-shot evaluation, leading models fail to detect errors in more than 50% of submissions flagged as incorrect by users.
However, when employing test-time scaling, success rates exceed 90%, although this approach incurs high computational costs. Notably, the best models can identify errors in over 5% of submissions that received full scores across approximately 30 problems, indicating that LLMs can provide valuable insights beyond traditional judging systems.