Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference
TL;DR
Majority voting is commonly used for aggregating answers from multiple samples in large language model (LLM) inference, but it overlooks valuable information. A new method called Propagational Proxy Voting (PPV) was developed to utilize additional signals from each sample.
✦ Why It Matters
Engineers can improve LLM inference accuracy by adopting delegation-based aggregation methods like PPV.
Key Takeaways
- Explore PPV for better aggregation in LLM tasks.
- Utilize letter entropy to inform voter weight decisions.
- Analyze reasoning geometry to improve consensus accuracy.
- Avoid reliance on majority voting in incoherent group scenarios.
How It Works
PPV operates by partitioning sampled responses into groups and calculating their letter-level semantic entropy and reasoning embedding centroids. It then uses these metrics to inform how much weight each voter retains on their own choice and how they distribute the remaining weight among peers.
This method allows PPV to effectively capture and utilize signals that majority voting typically ignores.