Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 20·5 min read
When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators
TL;DR
Large Language Models (LLMs) can effectively identify data quality issues, particularly when background knowledge is essential, but their performance varies by task. In e-commerce data quality tasks, LLMs showed limited advantage over rule-based methods in some cases, while excelling in others.
✦ Why It Matters
Evaluate your data quality tasks to determine if LLMs can provide significant advantages over traditional methods.
Key Takeaways
- Use LLMs for tasks needing background knowledge.
- Avoid relying on few-shot prompting without validation.
- Consider rule-based methods for tasks with strong lexical signals.
- Assess LLM consistency before deployment in critical applications.
Full Summary
Large Language Models (LLMs) are increasingly utilized for automatic detection of data quality issues, yet their consistency and effectiveness remain underexplored. This study evaluates an LLM's performance on two e-commerce data quality tasks: entity matching and brand mislabeling.
In entity matching, a rule-based approach achieved an F1 score of 0.950, closely matching the LLM's zero-shot performance of 0.948. However, a few-shot prompting method that seemed promising on a small sample resulted in a lower F1 score of 0.914 on a larger scale.
Conversely, in brand mislabeling detection, the LLM outperformed a naive rule-based method (F1 of 0.833 vs. 0.721) due to its ability to leverage background knowledge.
The LLM demonstrated high consistency, agreeing with itself 99.7% of the time across multiple runs. These findings indicate that LLMs are more beneficial for tasks requiring contextual understanding rather than those relying on straightforward lexical cues.