Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 20·5 min read
ReWEIGH the Evidence: Calibrating Token-Level Ordinal Visual Evidence to Mitigate Hallucinations in Large Vision-Language Models
TL;DR
ReWEIGH is a novel decoding intervention designed to reduce hallucinations in large vision-language models (LVLMs) by calibrating token-level visual evidence. It achieves up to a 21.3% reduction in hallucinated object mentions while maintaining or improving overall performance.
✦ Why It Matters
Implement ReWEIGH in your LVLMs to significantly reduce hallucinations in generated content.
Key Takeaways
- Integrate ReWEIGH to enhance the reliability of LVLM outputs.
- Utilize vocabulary ranks for better token evaluation.
- Monitor latency increases when applying ReWEIGH during inference.
- Test ReWEIGH across different model architectures for optimal performance.
How It Works
ReWEIGH operates by aggregating visual evidence from token states in LVLMs, which reflect how well the input image supports each token. It uses vocabulary ranks to create a scale-invariant basis for comparison, allowing the model to evaluate each token against a reference derived from unlabeled images.
During inference, it applies a bounded penalty to candidates that fall below their reference, effectively reducing hallucinations while maintaining performance.