Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 27·6 min read
Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregation
TL;DR
When large language models (LLMs) judge outputs for multiple users with conflicting preferences, current methods blend preference estimation and preference weighting together, creating unstable and unpredictable weight assignments. DecompR separates these two steps using counterfactual calibration to stabilize scoring.
✦ Why It Matters
Engineers can use DecompR to build more stable multi-user systems where conflicting preferences no longer cause unpredictable scoring shifts.
Key Takeaways
- Separate preference estimation from preference weighting to reduce score instability in multi-user systems.
- Apply counterfactual calibration when aggregating conflicting stakeholder preferences to stabilize LLM judge outputs.
- Monitor score volatility when stakeholder count increases; decomposition mitigates this scaling problem.
- Test DecompR on tasks where user preferences diverge significantly for maximum stability gains.
Full Summary
Multi-stakeholder alignment addresses a practical problem: how to generate one output that satisfies multiple users whose preferences conflict. Current LLM judges use holistic scoring—estimating and aggregating user satisfaction in a single step—which implicitly assigns unstable weights to different stakeholders.
The paper identifies weighting noise as the source of large score shifts when stakeholder satisfaction is dispersed, with instability increasing as stakeholder count grows. DecompR (Decomposed Counterfactual-Calibrated Ranking) separates utility estimation (measuring how much each user likes an output) from utility aggregation (combining those measurements into one score).
The method uses counterfactual calibration to stabilize implicit weights. Empirical and theoretical analysis demonstrates that decomposition reduces score volatility compared to holistic approaches, particularly in scenarios with high preference disagreement among stakeholders.