Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 2·5 min read
Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
TL;DR
Large language models (LLMs) significantly enhance conversational recommendation systems, with performance heavily influenced by retrieval methods and scoring protocols. A proprietary LLM reranker achieved an NDCG@10 score of 0.1497, outperforming non-LLM baselines by a notable margin.
✦ Why It Matters
Focus on optimizing candidate generation and scoring methods to enhance LLM performance in recommendation systems.
Key Takeaways
- Experiment with different candidate pool sizes for better LLM performance.
- Consider switching to collaborative-filtering candidates to boost NDCG scores.
- Adjust decoding temperature to explore its impact on model outputs.
- Report candidate generation and scoring methods as essential evaluation metrics.
Full Summary
Conversational recommender systems increasingly utilize large language models (LLMs) as rerankers, but their effectiveness is contingent on retrieval and inference strategies. In this study, various LLM rerankers were evaluated against collaborative-filtering and sequential baselines using the ReDial movie recommendation benchmark.
The best proprietary LLM reranker achieved an NDCG@10 score of 0.1497, compared to 0.0939 for the strongest non-LLM baseline. Notably, switching from semantic to collaborative-filtering candidates improved performance by over 50%.
Additionally, increasing the decoding temperature from 0 to 1.0 affected the Jaccard distance but had minimal impact on NDCG@10 for the best reranker. These findings emphasize the importance of candidate generation and scoring policies in evaluating LLM performance, suggesting they should be reported as critical factors in future research.