NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·May 27·8 min read
Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines
TL;DR
Most alignment tuning research focuses on optimization algorithms while treating training data construction as secondary. This survey reframes alignment as a data pipeline design problem, decomposing it into three stages: response synthesis (generating candidate outputs), preference evaluation (scoring outputs), and preference instantiation (converting scores to training signals).
✦ Why It Matters
Engineers can systematically design alignment data pipelines by understanding how response synthesis, preference evaluation, and instantiation choices interact.
Key Takeaways
- Treat alignment data construction as explicit pipeline design, not implicit implementation detail.
- Decompose alignment into three stages: synthesis, evaluation, and instantiation for systematic analysis.
- Evaluate how choices at each pipeline stage affect downstream model behavior and training efficiency.
- Use the unified taxonomy to identify gaps and compare alignment methods across different approaches.
Full Summary
Alignment tuning makes large language models (LLMs) behave according to human values and intentions. Prior work emphasizes optimization objectives—the mathematical formulas guiding model training—but largely ignores how alignment training data is actually constructed.
This survey adopts a data-centric perspective, treating alignment as a pipeline design problem with three interacting stages. Response synthesis generates multiple candidate outputs for a given input.
Preference evaluation scores or ranks these outputs based on quality criteria. Preference instantiation converts these evaluations into training signals the model learns from.
By organizing existing alignment methods through this three-stage framework, the authors create a unified taxonomy that reveals how different approaches make different choices at each stage, enabling systematic comparison and identifying gaps in current practices.