Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)
TL;DR
Large language models (LLMs) often show diminishing returns in performance when answering questions, measured by pass@k. To address this, Reset-and-Discard (ReD) was developed as a query method that enhances coverage@cost, which is the number of unique questions answered per budget spent.
✦ Why It Matters
Engineers can use ReD to optimize LLM performance and reduce costs in practical applications.
Key Takeaways
- Implement Reset-and-Discard to enhance LLM efficiency.
- Measure performance using coverage@cost for budgeted tasks.
- Utilize ReD to predict savings in attempts for LLMs.
- Apply findings to improve cost-effectiveness in AI applications.
Full Summary
Large language models (LLMs) are evaluated on their ability to answer questions correctly, typically using a metric called pass@k, which measures the probability of a correct answer in multiple attempts. However, at a fixed budget, a more relevant metric is coverage@cost, which assesses how many unique questions are answered relative to the total attempts made.
The Reset-and-Discard (ReD) method was introduced to improve coverage@cost by strategically resetting and discarding attempts that are less likely to yield new answers. Through experiments on three benchmarks—coding (HumanEval), math (GSM8K), and reasoning (MMLU-Pro)—ReD was shown to reduce the number of attempts and associated costs while maintaining or improving coverage.
The findings indicate that ReD can also predict savings in attempts based on the pass@k metric, making it a versatile tool for optimizing LLM inference. This method remains effective even with imperfect verification processes, outperforming existing allocation strategies.