Dynamics-Aware Preference Optimization for Vision-Language Models

Preference-based finetuning of vision-language models (VLMs) is notoriously unstable, i.e., trivially wrong negatives inject uninformative gradients that distort optimization and degrade calibration. This work revisits this issue through the lens of learning dynamics and identifies a core pathology, i.e., the squeezing effect, where easy negatives retain large, misaligned gradients despite negligible loss. To address this, we propose Cooling-Weighted Direct Preference Optimization (CW-DPO), a two-stage framework that smooths and then stabilizes the alignment process. Stage 1 employs a constrained SFT phase with low-weight "gentle negatives" to regularize overconfident distributions and flatten the loss landscape. Stage 2 introduces a competence-aware cooling weight that adaptively scales negative gradients according to the model's average per-token log-probability, suppressing uninformative updates while emphasizing hard, on-policy contrasts. This dynamics-aware weighting effectively mitigates the squeezing effect and enables smoother convergence. Extensive and comprehensive results on the mainstream benchmarks, i.e., COCO, Flickr30k, NoCaps, MMMU, and MMBench1.1, our CW-DPO achieves stateof-the-art performance, e.g., +3.4 CIDEr over PPO and +2.4% absolute accuracy on MMMU, while improving calibration and halving convergence steps. This justifies that smoothing before cooling constitutes a simple yet general principle for robust VLM preference optimization. https://github.com/jushengzhang/Dynamics-Aware-Preference-Optimization