Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 25·8 min read
PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
TL;DR
Large language models (LLMs) struggle with pricing negotiations when buyer preferences—like willingness to pay and negotiation style—remain hidden from the seller. PrefBench is a simulator-based benchmark that pairs LLM sellers with simulated buyers in vehicle-customization pricing scenarios, constraining agents to return structured JSON actions.
✦ Why It Matters
Engineers building LLM-based negotiation or pricing systems can identify that high agreement rates mask weak economic performance without proper profit-focused evaluation.
Key Takeaways
- Utilize PrefBench to benchmark LLM performance in pricing negotiations.
- Explore the impact of hidden buyer preferences on negotiation outcomes.
- Compare LLM strategies against heuristic methods for better insights.
- Focus on improving profit-sensitive decision-making in LLM agents.
How It Works
PrefBench simulates personalized pricing negotiations by pairing a seller with a buyer whose preferences are hidden. The seller has access to public persona descriptors and negotiation history, while latent variables influence the buyer's valuation and behavior.
The LLM must generate actions in a structured JSON format, adhering to a fixed boundary of hidden information.