SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-rewards

Building upon large language models (LLMs), recent large multimodal models (LMMs) unify cross-modal understanding and generation into a single framework. However, LMMs still struggle to achieve accurate vision-language alignment, prone to generating text responses contradicting the visual input or failing to follow the text-to-image prompts. To alleviate these issues, a promising line of research explores improving LMMs in the post-training stage. However, most existing solutions rely on external supervision (e.g., human annotations, reward models) and are typically tailored to only unidirectional tasks, i.e., optimizing either vision understanding or generation. In this work, based on the observation that understanding and generation are naturally inverse dual tasks, we propose SUDER (Self-improving Unified LMMs with Dual sElf-Rewards), a framework reinforcing the understanding and generation capabilities of LMMs with a self-supervised dual reward mechanism. SUDER leverages the inherent duality between understanding and generation to provide self-supervised optimization signals for each other. Specifically, we sample multiple outputs for a given input, then reverse the input-output pairs to compute the dual likelihood within the model as self-rewards for optimization. Extensive experimental results on visual understanding and generation benchmarks demonstrate that our method can effectively enhance the performance of the LMM without any external supervision, especially achieving remarkable improvements in text-to-image tasks.