Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 25·6 min read
CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs
TL;DR
Large vision-language models (systems combining image and text understanding) hallucinate objects that aren't present when language patterns override weak visual signals. CHASD, a Contrastive Hallucination-Aware Step-wise Decoding method, selectively applies contrastive decoding—comparing predictions from original and perturbed images—only when model confidence is low.
✦ Why It Matters
Engineers can deploy CHASD at inference time without retraining to reduce hallucinations in vision-language systems while preserving computational efficiency.
Key Takeaways
- Hallucination risks are token-specific and transient; selective calibration outperforms global approaches.
- Confidence gating activates contrastive branches only when model uncertainty exceeds threshold.
- Attention-guided perturbations target salient visual regions instead of global image modifications.
- CHASD requires no model retraining and works as an inference-time wrapper.
Full Summary
Large vision-language models (LVLMs) excel at multimodal reasoning but suffer from object hallucinations when language priors dominate insufficient visual evidence. Existing training-free contrastive decoding methods mitigate this by comparing predictions from original and perturbed visual inputs, but they either apply global perturbations that destroy useful visual information or require expensive negative-branch computations at every decoding step.
CHASD introduces an uncertainty-driven confidence gate that activates contrastive calibration only when the maximum probability of the next token falls below a threshold, using attention-guided localized perturbations of salient visual regions rather than global image modifications. This on-demand calibration approach reduces unnecessary forward passes while preserving the original prediction distribution for high-confidence tokens.
Experiments across POPE, AMBER, MME, MMHal-Bench, and CHAIR benchmarks demonstrated CHASD improved hallucination-related metrics compared to strong training-free baselines while maintaining competitive inference efficiency.