NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 28·6 min read
MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation
TL;DR
Large language models used as judges to evaluate code correctness often make inconsistent decisions. MCTS-Judge applies Monte Carlo Tree Search (a decision-exploration algorithm) at test time to improve judgment quality.
✦ Why It Matters
Engineers can deploy more reliable code evaluation systems by using test-time search instead of single-pass LLM judgments.
Key Takeaways
- Apply MCTS at inference time to improve LLM judgment consistency on code evaluation tasks.
- Use multiple reasoning paths and aggregate results instead of relying on single LLM verdicts.
- Test-time scaling trades computation for accuracy without retraining the underlying model.
- Particularly effective for ambiguous or complex code correctness cases where single-pass judgment fails.
Full Summary
Evaluating whether code is correct is challenging for large language models (LLMs)—AI systems trained on text that can generate and reason about code. Current approaches use LLMs directly as judges, but they produce inconsistent verdicts on identical problems.
MCTS-Judge introduces Monte Carlo Tree Search (MCTS), an algorithm that explores multiple reasoning paths by sampling and evaluating different decision branches, applied during inference time (test-time scaling). Rather than making a single judgment, MCTS-Judge generates multiple evaluation attempts and aggregates them to reach a more robust conclusion.
This approach mirrors how humans might reconsider a problem from different angles before deciding. Results demonstrate improved accuracy in code correctness evaluation, with the method particularly effective on ambiguous or complex cases where single-pass judgment fails.