Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·Aug 6·5 min read
FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents
TL;DR
FinPerMA is a new benchmark designed to evaluate how well large language model (LLM) agents can maintain personalized memory over time, particularly in financial advising. Initial results show that current LLMs struggle to accurately integrate user preferences, achieving only about 47% accuracy overall.
✦ Why It Matters
Evaluate your LLM's personalized memory capabilities using the FinPerMA benchmark to identify areas for improvement.
Key Takeaways
- Test LLMs against the FinPerMA benchmark for personalized memory evaluation.
- Focus on integrating user preferences rather than just factual retention.
- Consider using retrieval methods for better performance in memory tasks.
- Analyze how well your LLM adapts to significant user events.
How It Works
FinPerMA generates personalized memory scenarios using deterministic rules based on financial events, allowing LLMs to narrate user trajectories. This structured approach helps assess how well agents can adapt their memory to reflect changes in user preferences over time.