Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 20·6 min read
Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application
TL;DR
A benchmark was developed to evaluate open-source models for structured information extraction from student applications, revealing that most configurations performed poorly. Only 4 out of 35 configurations achieved satisfactory performance, highlighting challenges in using these models in high-risk public sector applications.
✦ Why It Matters
Evaluate your current document processing tools against open-source models to ensure compliance with high-risk standards.
Key Takeaways
- Assess the performance of open-source OCR and LLM models for your applications.
- Focus on input quality to improve extraction outcomes.
- Consider VLMs for better performance in document processing tasks.
- Regularly benchmark your models against new open-source solutions.
Full Summary
Structured information extraction from unstructured documents is vital for digital transformation, especially in high-risk public sector applications. This study benchmarks open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) on a complex task: processing student applications for an international study program.
The evaluation found that VLMs generally outperformed OCR+LLM combinations, but only 4 of 35 configurations achieved F1 scores above 0.5, indicating limited reliability in zero-shot settings. Approximately 75% of configurations scored below 0.25, and while larger models tended to perform better, the relationship was non-linear.
Input quality, particularly how well OCR preserves document structure, was identified as a critical factor affecting performance.