Focus on What Matters: Enhancing Medical Vision-Language Models with Automatic Attention Alignment Tuning

Medical Large Vision-Language Models (Med-LVLMs) often exhibit suboptimal attention distribution on visual inputs, leading to hallucinated or inaccurate outputs. Existing mitigation methods primarily rely on inference-time interventions, which are limited in attention adaptation or require additional supervision. To address this, we propose A 3 TUNE, a novel fine-tuning framework for Automatic Attention Alignment Tuning. A 3 TUNE leverages zeroshot weak labels from SAM, refines them into prompt-aware labels using BiomedCLIP, and then selectively modifies visually-critical attention heads to improve alignment while minimizing interference. Additionally, we introduce a A 3 MOE module, enabling adaptive parameter selection for attention tuning across diverse prompts and images. Extensive experiments on medical VQA and report generation benchmarks show that A 3 TUNE outperforms state-ofthe-art baselines, achieving enhanced attention distributions and performance. 1