Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 4·5 min read
DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents
TL;DR
DuplexSpeechBench-IFEval (DSB-IFEval) was developed to assess how well full-duplex voice agents follow implicit instructions during real-time interactions. The evaluation revealed significant performance variations based on whether behaviors were explicitly stated or inferred from personas.
✦ Why It Matters
Evaluate your voice agent's ability to follow implicit instructions to enhance user interaction quality.
Key Takeaways
- Test voice agents with DSB-IFEval to assess implicit instruction adherence.
- Focus on persona conditioning to improve conversational behavior.
- Analyze trade-offs between explicit and inferred instructions in your models.
- Ensure your system can handle conflicting directives effectively.
Full Summary
Full-duplex voice agents, which can listen and speak simultaneously, face challenges in managing conversations effectively. Existing benchmarks often rely on explicit instructions, but real-world applications require agents to infer appropriate behaviors from roles or personas.
DSB-IFEval introduces a comprehensive evaluation framework with 1,038 test cases across eight assistant roles, assessing five instruction-following protocols. Key metrics include the Instruction Adherence Score (IAS) for floor management and the Persona Adherence Score (PAS) for content consistency.
Results showed that models like F-Actor and PersonaPlex struggled with persona-only conditioning, with adherence dropping by 9.7% and 4.5%, respectively. In contrast, models like GPT-Realtime maintained strong persona adherence but exhibited limited adaptability in floor behavior.
These findings highlight the distinct challenges of inferring behaviors, executing them timely, and resolving conflicting instructions in voice agents.