Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Jul 29·5 min read
CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
TL;DR
CoRT introduces a novel method for token-level credit allocation in rubric-based reinforcement learning, enhancing the training of language models. It achieves an average performance improvement of 4.4 percentage points over traditional response-level methods without requiring an auxiliary scoring model.
✦ Why It Matters
Implement CoRT to improve token-level credit allocation in your language model training today.
Key Takeaways
- Adopt CoRT for improved token credit allocation in language models.
- Utilize counterfactual replay to enhance training efficiency.
- Avoid the complexity of auxiliary scoring models.
- Achieve better performance metrics with existing GRPO frameworks.
How It Works
CoRT employs counterfactual replay to evaluate the same response under two different prompts: one with rubric conditions and one without. By comparing the log-likelihood of tokens in these contexts, CoRT generates contrasts that indicate how much each token's likelihood depends on the rubric.
These contrasts are then normalized and used to adjust the rewards for individual tokens, allowing for a more nuanced credit allocation that reflects their contribution to the overall response.