FINER: MLLMs Hallucinate under Fine-grained Negative Queries

Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FInegrained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and "what" questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at https://explainableml.github.io/finer-project/. Can you see the cat in this image? Can you see the wolf in this image? Can you see the cat with predominantly white coat featuring black and grey markings in this image? Can you see the cat with predominantly brown coat featuring orange and pink markings in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head tilted backward in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with drooping ears in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears , that is sitting on the chair in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears , that is sitting below the chair in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears , that is sitting on the chair in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears , that is sitting on the sofa in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears , that is sitting on the chair with primarily blue and white in color exhibiting signs of wear in this image? Can you see the cat with predominantly white coat featuring black and grey markings, with its head turned downwards, with perked ears , that is sitting on the chair with largely purple and orange in color indicating heavy use in this image? Can you see the cat in this image? Can you see the wolf? Can you see the ca and grey markings i Can you see the cat with predominantly brown coat? Can you see the cat w image? Can you see the cat with …, with its head tilted backward? Can you see the cat with …, …, with perked ears in th Can you see the cat with …, …, with drooping ears? Can you see the cat with …, …, …, that is sitting on the this image? Can you see the cat with …, ..., …, that is sitting below the chair? Can you see the cat with …, …, …, that is sitting on the this image?

Can you see the cat with …, …, …, that is sitting on the sofa?

Can you see the cat with …, …, …, that is … with prima and white in color in this image?