Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?

Improving semantic grounding in Vision-Language Models (VLMs) often involves collecting domain-specific training data, refining the network architectures, or modifying the training recipes. In this work, we venture into an orthogonal direction and explore self-correction in VLMs focusing on semantic grounding. We find that VLMs can correct their own semantic grounding mistakes when properly prompted and framed for the task, without any fine-tuning or even access to oracle feedback. We also introduce a self-correction framework in an iterative setting which consistently improves performance across all models investigated. Overall, we show that iterative self-correction consistently improves VLM performance in semantic grounding by up to 8.4 accuracy points across all models investigated, without requiring fine-tuning, additional architectural changes, or external data. Our exploration of self-correction also reveals that, even after several rounds of feedback, strong models like GPT-4V and GPT-4o retain limited capability in leveraging oracle feedback, suggesting promising directions for further research. User: What is inside the specified region? VLM: It is a modern style armchair. Agent: It is a modern style dinning table in a living room. Agent: It is a modern style armchair. System: A verifier disagrees with your prediction. Review the image again. System: Does the specified region contain a modern style dining table in a living room? Verifier: No the specified region does not. Automated feedback-based verification System: Does the specified region contain a modern style armchair? Verifier: Yes.