Multimodal Reasoning with Explicit Reasoning Patterns and Rewards

Large language models (LLMs) have demonstrated explicit reasoning capabilities such as reflection, verification, and exploration via reinforcement learning techniques such as GRPO. However, multimodal large language models (MLLMs) are still lacking in such reasoning capabilities despite their impressive performance in understanding visual and language data. We design RLRR, a novel reinforcement learning framework that guides multimodal reasoning by explicitly rewarding interpretable reasoning behaviours such as clarification, verification, and exploration. RLRR acquires the multimodal reasoning capabilities from three perspectives. First, it builds up the base reasoning capabilities via supervised fine-tuning, where a high-quality benchmark is constructed by collecting desired reasoning patterns from advanced reasoning models. Second, it enhances the MLLM base reasoning capabilities by introducing explicit reasoning rewards that reward the responses demonstrating the desired reasoning patterns. Third, we design a new reward to punish the model when it outputs semantically similar verbose content, enabling the model to adaptively output complex and brief reasoning for difficult and simple problems, respectively. Extensive experiments over multiple multimodal reasoning benchmarks show that the proposed RLRR achieves superior performance in guiding MLLMs on structured multimodal reasoning.