Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 31·5 min read
Program Learning with Verifiable Rewards: Symbolic Backpropagation for Post-Training LLMs
TL;DR
Program Learning with Verifiable Rewards (PLVR) introduces a method for post-training language models that allows reasoning to be externalized as explicit programs. This approach outperforms traditional reinforcement learning methods, achieving an average improvement of 27.8 points on benchmark tasks.
✦ Why It Matters
Implement PLVR to enhance your language model's reasoning capabilities without extensive retraining.
Key Takeaways
- Explore PLVR for externalizing reasoning in language models.
- Utilize symbolic backpropagation for efficient program learning.
- Leverage a single primitive library for multiple tasks.
- Achieve significant performance gains over traditional reinforcement learning.
Full Summary
PLVR addresses the limitations of traditional supervised finetuning and reinforcement learning, which embed reasoning capabilities within model weights, making them opaque and non-transferable. Instead, PLVR externalizes reasoning as explicit programs derived from input-output examples, utilizing a technique called symbolic backpropagation.
This method computes loss at the output and propagates required input types backward through a type inference process, akin to the chain rule in calculus. On benchmarks like LiveCodeBench v6 and Tau2Bench, PLVR demonstrated a significant performance boost, outperforming reinforcement learning by an average of 27.8 points for 30B base models.
Notably, the method requires only 100 examples for new tasks, leveraging a single primitive library across multiple benchmarks. The study also highlights that the backward pass in symbolic backpropagation is crucial for its success, rather than the type system alone.