ACL Findings2026
MENTOR: Efficient Autoregressive Image Generation with Balanced Multimodal Control
Haozhe Zhao, Zefan Cai, Shuzheng Si, Liang Chen, Jiuxiang Gu, Wen Xiao, Minjia Zhang, Junjie Hu
Abstract
Recent text-to-image models achieve impressive visual quality but still face challenges in precise controllability, balancing multimodal inputs, and high training cost for multimodal image generation. To address these limitations, we propose M ENTOR , an autoregressive (AR) framework with a two-stage training paradigm for controllable multimodal image generation: (1) a multimodal alignment stage that establishes robust pixel and semantic-level alignment between inputs and generated to-kens, followed by (2) a multimodal instruction tuning stage that balances the model’s integration of multimodal inputs and enhances generation controllability. Extensive experiments on DreamBench++ and DreamBench demonstrate that, despite modest model size and training resources, M ENTOR achieves a strong balance between textual and visual guidance for controllable image generation, delivering competitive performance at significantly lower computational cost compared to leading baselines. Moreover, our approach attains superior image reconstruction fidelity, broad adaptability across different tasks, and training efficiency.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Adding Conditional Control to Text-to-Image Diffusion Models cites Zero-Shot Text-to-Image Generation
- Adding Conditional Control to Text-to-Image Diffusion Models cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Zero-Shot Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
Builds on28
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations
- Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
Related papers
- DreamBench++: A Human-Aligned Benchmark for Personalized Image GenerationYuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi et al.ICLR 2025
- MMaDA-Parallel: Multimodal Large Diffusion Language Models for Thinking-Aware Editing and GenerationYe Tian, Ling Yang, Jiongfan Yang, Anran Wang et al.ICLR 2026 · 8 citations
- Jointly Training Large Autoregressive Multimodal ModelsEmanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan et al.ICLR 2024 · 35 citations
- AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image GenerationLianyu Pang, Jian Yin, Baoquan Zhao, Feize Wu et al.NeurIPS 2024 · 18 citations
- ORION: Decoupling and Alignment for Unified Autoregressive Understanding and GenerationTaihang Hu, Mengting Chen, Jinsong Lan, Xiaoyong Zhu et al.ICLR 2026
