Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models

Large Vision-Language Models (LVLMs) have achieved strong performance on visionlanguage tasks, particularly Visual Question Answering (VQA). While prior work has explored unimodal biases in VQA, the problem of selection bias in Multiple-Choice Question Answering (MCQA), where models may favor specific option tokens (e.g., "A") or positions, remains underexplored. In this paper, we investigate both the presence and nature of selection bias in LVLMs through fine-grained MCQA benchmarks spanning easy, medium, and hard difficulty levels, defined by the semantic similarity of the options. We further propose an inference-time logit-level debiasing method that estimates an ensemble bias vector from general and contextual prompts and applies confidence-adaptive corrections to the model's output. Our method mitigates bias without retraining and is compatible with frozen LVLMs. Extensive experiments across several state-ofthe-art models reveal consistent selection biases that intensify with task difficulty, and show that our mitigation approach significantly reduces bias while improving accuracy in challenging settings. This work offers new insights into the limitations of LVLMs in MCQA and presents a practical approach to improve their robustness in fine-grained visual reasoning. Datasets and code are available at: https://github.com/ Atabuzzaman/Selection-Bias-of-LVLMs * The proposed bias mitigation method was fully created and implemented by this author. Yellow-headed Blackbird B. This image shows a Yellow-headed Blackbird bird which has a black body, bright yellow head… C. This is a Rolls-Royce Phantom Drophead Coupe Convertible 2012 car which has a distinctive… D. This image shows a hot and sour soup food item which has a rich broth, egg strands, tofu, and … A. This image shows a Yellow-headed Blackbird bird which has a black body, bright yellow head… B. This image shows a Brown Creeper bird with small, dark eyes, a slender, upturned beak, and… C. This image shows a Florida Jay bird which has a small, blue-gray body, a white chest and belly… D. This image shows a Tree Sparrow bird which has a brown and white body, a chestnut cap … A. This image shows a Rusty Blackbird bird which has rusty-brown plumage in non-breeding… B This image shows a Red-winged Blackbird bird which has a black body, distinctive red and… C. This image shows a Brewer Blackbird bird which has a sleek black body, iridescent feathers… D. This image shows a Yellow-headed Blackbird bird which has a black body, bright yellow head… A. This is a Spyker C8 Coupe 2009 car which has a distinctive propeller logo, sleek aerodynamic … Hard Easy Medium