Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Jul 27·5 min read
Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
TL;DR
A novel approach integrates documents directly into the weights of a 4-bit Gemma-4-e4b model using LoRA, enhancing closed-book question answering accuracy. Key findings reveal that data quality significantly impacts performance, achieving an accuracy increase from 57.7% to 85.7% with improved training data.
✦ Why It Matters
Focus on improving training data quality to enhance closed-book QA model performance immediately.
Key Takeaways
- Prioritize high-quality training data for better model accuracy.
- Ensure adapter capacity is adequate before training.
- Consider using LoRA for efficient model adaptation.
- Evaluate performance against established retrieval methods.
How It Works
The approach integrates documents directly into the model's weights using LoRA, allowing the model to answer questions without retrieving external information. This method leverages low-rank adaptation to efficiently internalize knowledge from the training documents.