Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 25·8 min read
LFRAG: Layout-oriented Fine-grained Retrieval-Augmented Generation on Multimodal Document Understanding
TL;DR
Existing systems that augment language models with external documents retrieve entire pages, missing fine-grained semantic structure in visually complex documents. LFRAG shifts retrieval from page-level to block-level (smaller semantic units) using layout segmentation and a semantic-layout fusion encoder combining text and visual context.
✦ Why It Matters
Engineers can build more accurate and efficient document-based AI systems by adopting block-level retrieval instead of page-level approaches.
Key Takeaways
- Shift document retrieval from page-level to block-level granularity for better semantic precision.
- Use layout segmentation to identify coherent semantic units within visually complex documents.
- Integrate visual positioning and text semantics via cross-attention in encoder design.
- Expect 7.20% accuracy gains and 73% token reduction versus page-level baselines.
Full Summary
Multimodal Retrieval-Augmented Generation (RAG)—a technique where language models query external documents to answer questions—typically retrieves entire pages, losing semantic and layout structure in visually rich documents like PDFs or forms. LFRAG introduces block-level retrieval, breaking documents into semantically coherent chunks smaller than pages.
The framework uses layout segmentation to identify these blocks and a semantic-layout fusion encoder that combines local text meaning with global visual positioning via cross-attention mechanisms. The authors created LFDocQA, a large-scale benchmark with fine-grained block-level annotations across diverse document types, enabling rigorous evaluation.
Experiments show LFRAG achieves state-of-the-art retrieval performance, outperforms best baselines by 7.20% in answer accuracy, and reduces token consumption by 73.07%, making it more efficient for downstream generation tasks.