Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 25·5 min read
PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models
TL;DR
PatchGate is a training-free framework that enhances image captioning in Vision-Language Models by improving object mention accuracy. It increases visible-object coverage by 13.4% while reducing hallucinations by 12.0%.
✦ Why It Matters
Implement PatchGate to enhance your image captioning systems without extensive retraining.
Key Takeaways
- Use PatchGate to improve image captioning accuracy.
- Leverage Visual Evidence eXtraction for intrinsic object identification.
- Apply Visual-Evidence Inclusion-Exclusion Decoding to refine outputs.
- Test on AMBER to evaluate performance improvements.
How It Works
PatchGate operates in two stages. First, Visual Evidence eXtraction (VEX) analyzes the latter layers of a frozen VLM to extract object evidence from image patches, creating an object set without needing prompts.
Next, Visual-Evidence Inclusion-Exclusion Decoding (VIED) adjusts the model's decoding process, enhancing the likelihood of under-represented objects while reducing the prominence of over-represented ones.