Detection and Correction of Object Hallucination using Attention Map and Gradient Information in LVLMs

DOI

Bibliographic Information

Other Title
  • LVLMsにおけるAttention Mapと勾配情報を活用したObject Hallucinationの検出と修正

Description

<p>Inspired by the superior language processing capabilities of Large Language Models (LLMs), there has been a recent push to develop Large Vision Language Models (LVLMs) that incorporate powerful LLMs to enhance performance on complex multimodal tasks. However, these LVLMs face issues with Object Hallucination, where they inaccurately recognize and describe objects that do not exist in the image or misrepresent the relationships between objects. To address this problem, we propose a framework that detects and corrects Object Hallucination. This framework identifies and detects the specific parts of an image that cause Object Hallucination based on Attention Maps and gradient information within the LVLMs, and then makes corrections. Through experiments, we have verified that our proposed method reduces the occurrence of Object Hallucination using multiple quantitative metrics.</p>

Journal

Keywords

Details 詳細情報について

Report a problem

Back to top