MENTOR: Efficient Autoregressive Image Generation with Balanced Multimodal Control

Recent text-to-image models achieve impressive visual quality but still face challenges in precise controllability, balancing multimodal inputs, and high training cost for multimodal image generation. To address these limitations, we propose M ENTOR , an autoregressive (AR) framework with a two-stage training paradigm for controllable multimodal image generation: (1) a multimodal alignment stage that establishes robust pixel and semantic-level alignment between inputs and generated to-kens, followed by (2) a multimodal instruction tuning stage that balances the model’s integration of multimodal inputs and enhances generation controllability. Extensive experiments on DreamBench++ and DreamBench demonstrate that, despite modest model size and training resources, M ENTOR achieves a strong balance between textual and visual guidance for controllable image generation, delivering competitive performance at significantly lower computational cost compared to leading baselines. Moreover, our approach attains superior image reconstruction fidelity, broad adaptability across different tasks, and training efficiency.