Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards
Seungwook Kim, Minsu Cho
Abstract
Text-to-image generation powers content creation across design, media, and data augmentation. Post-training of text-to-image generative models is a promising path to improve human preference alignment, factuality, and aesthetics. We introduce SOLACE (Self-Originating LAtent Confidence Estimation), a post-training framework that replaces external reward supervision with an internal self-confidence signal: we re-noise the model's own outputs and measure how accurately it recovers the injected noise, treating low reconstruction error as high self-confidence. SOLACE converts this intrinsic signal into scalar rewards for reinforcement learning, requiring no external reward models, annotators, or preference data. By reinforcing high-confidence generations, SOLACE delivers consistent gains in compositional generation, text rendering, and text-image alignment. Integrating SOLACE with external rewards yields complementary improvements while alleviating reward hacking.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on56
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
Related papers
- Understanding Reward Hacking in Text-to-Image Reinforcement LearningYunqi Hong, Kuei-Chun Kao, Hengguang Zhou, Cho-Jui HsiehCVPR Findings 2026
- Alignment-Guided Score Matching for Text-to-Image Alignment in Diffusion ModelsJaa-Yeon Lee, Yeobin Hong, Taesung Kwon, Jong Chul YEICML 2026
- SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward ModelsJiesong Lian, Ruizhe Zhong, Zixiang Zhou, Xiaoyue Mi et al.CVPR 2026 · 3 citations
- Style-GRPO: Semantic-Aware Preference Optimization for Image Style Transfer Guided by Reward ModelingJianbin Zhao, Chaoran Feng, Miao Yu, Yingtao Li et al.CVPR 2026
- Enhancing Reward Models for High-Quality Image Generation: Beyond Text-Image AlignmentYing Ba, Tianyu Zhang, Yalong Bai, Wenyi Mo et al.ICCV 2025 · 13 citations
