NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 21·6 min read
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
TL;DR
Evaluating code quality automatically is hard because judges (AI models) often disagree on which code is better and why. CriterAlign is a method that aligns an AI judge's reasoning with explicit evaluation criteria (like readability, efficiency, correctness) by training it to explain decisions using those specific criteria.
✦ Why It Matters
Engineers building code review automation or LLM-based development tools can use criterion-centric alignment to make AI judges more reliable and interpretable.
Key Takeaways
- Train models to explicitly reference evaluation criteria when explaining code judgments.
- Measure both preference accuracy and reasoning alignment to criteria separately.
- Use criterion-centric explanations to improve interpretability in automated code review.
- Apply rationale alignment to make AI code judges auditable and trustworthy.
Full Summary
When AI judges code quality, it often produces correct rankings but with reasoning that doesn't match how humans actually evaluate code. CriterAlign addresses this gap by implementing criterion-centric rationale alignment—a training approach where models learn to explain their code preferences by explicitly referencing evaluation criteria (like readability, efficiency, or correctness).
The method works by fine-tuning language models to generate justifications that map directly to predefined judgment criteria rather than producing generic explanations. During evaluation, researchers measured both whether the model's final preference matched human judgment and whether the reasoning cited relevant criteria.
Results showed CriterAlign improved the consistency between model explanations and the actual criteria driving decisions, making AI code judges more interpretable and trustworthy.