Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·11d ago·5 min read
Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening
TL;DR
A new metric, the Voice Retention / Representation Ratio, reveals that large language model (LLM) summaries often filter out critical employee feedback, favoring popular sentiments. This study highlights significant biases in how employee voices are represented in organizational summaries.
✦ Why It Matters
Evaluate your feedback summarization processes to ensure critical employee voices are not overlooked.
Key Takeaways
- Implement the Voice Retention metric in feedback processes.
- Analyze the prevalence of themes in employee feedback.
- Ensure critical feedback is not filtered out in summaries.
- Consider bilingual responses to improve representation.
Full Summary
Organizations increasingly use large language models (LLMs) to summarize employee feedback, but this process can mute important voices. The study introduces the Voice Retention / Representation Ratio metric to assess representational bias in summarization.
Analyzing 2,586 bilingual (English/German) responses from a global professional service company, findings show that employees are 82 times more likely to withhold praise than to provide criticism. Despite criticism being retained, concerns expressed once are often omitted 86% of the time, particularly affecting shorter and German-only responses.
The analysis indicates that sentiment does not independently influence retention; rather, the prevalence of themes drives the outcomes. A targeted prompt can recover named themes, but broader concerns remain unaddressed.
This research contributes a new metric and empirical evidence to understand voice retention in organizational contexts.