Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
Data Science
Towards Data Science·Aug 31·5 min read
Your LLM Can Return Perfect JSON and Still Be Wrong
TL;DR
Large Language Models (LLMs) can generate syntactically correct JSON outputs but still produce incorrect or misleading information due to the quality of input data. This highlights the importance of data integrity in AI applications.
✦ Why It Matters
Ensure data quality and validation checks in your AI applications to avoid misleading outputs.
Key Takeaways
- Use nullable fields to prevent models from inventing data.
- Implement evidence fields to verify data against source text.
- Shift focus from valid JSON to data truthfulness.
- Limit retries on failed extractions to avoid unnecessary API calls.
Full Summary
A payment processing pipeline using Structured Outputs encountered mismatches in transaction records, where amounts and senders matched, but dates were incorrect. The issue arose because the model filled in a required date field with the current date when no date was present in the source message.
This highlighted a flaw in treating valid JSON as a success metric, as it allowed for plausible but incorrect data. To address this, the author suggests using nullable fields to prevent the model from inventing values and implementing evidence fields to track the source of each value.
This approach ensures that the data extracted is both accurate and verifiable, shifting the focus from merely achieving a valid schema to ensuring the truthfulness of the data.