CVPR Findings2026
VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal Reinforcement Learning
Zengjie Hu, Jiantao Qiu, Tianyi Bai, Haojin Yang, Binhang Yuan, Qi Jing, Conghui He, Wentao Zhang
Abstract
Group-based policy optimization methods like GRPO and GSPO have become standard for training multimodal models, leveraging group-wise rollouts and relative advantage estimation. However, they suffer from a critical gradient vanishing problem when all responses within a group receive identical rewards, causing advantage estimates to collapse and training signals to diminish. Existing attempts to mitigate this issue fall into two paradigms: filtering-based and sampling-based methods. Filtering-based methods first generate rollouts broadly and then retroactively filter out uninformative groups, leading to substantial computational overhead. Sampling-based methods proactively select effective samples before rollout but rely on static criteria or prior dataset knowledge, lacking real-time adaptability. To address these issues, we propose VADE, a Variance-Aware Dynamic sampling framework via online sample-level difficulty Estimation. Our framework integrates three key components: online sample-level difficulty estimation using Beta distributions, a Thompson sampler that maximizes information gain through the estimated correctness probability, and a two-scale prior decay mechanism that maintains robust estimation under policy evolution. This three components design enables VADE to dynamically select the most informative samples, thereby amplifying training signals while eliminating extra rollout costs. Extensive experiments on multimodal reasoning benchmarks show that VADE consistently outperforms strong baselines in both performance and sample efficiency, while achieving a dramatic reduction in computational overhead. More importantly, our framework can serves as a plug-and-play component to be seamlessly integrated into existing group-based RL algorithms. Code are publicly released at https://vade-
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Builds on10
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningAhmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty et al.ACL Findings 2022 · 389 citations
Related papers
- DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant AdvantageHaowen Gao, Zhenyu Zhang, Liang Pang, Fangda Guo et al.ICLR 2026 · 3 citations
- Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable RewardsHieu Trung Nguyen, Bao Nguyen, Wenao Ma, Yuzhi Zhao et al.ICLR 2026 · 19 citations
- PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic ReasoningWenfeng Feng, Penghong Zhao, Guochao Jiang, Chuzhan Hao et al.ACL Findings 2026
- MMR1: Enhancing Multimodal Reasoning with Variance-Aware SamplingSicong Leng, Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Yuming Jiang, Hang Zhang, Xin Li, Deli Zhao, Wei Lu, Yu Rong, Aixin Sun, Shijian LuCVPR Findings 2026
- Group Distributionally Robust Optimization-Driven RL for LLM ReasoningKishan Panaganti, Zhenwen Liang, Wenhao Yu, Haitao Mi et al.ICML 2026 · 5 citations
