Lune

CVPR Findings2026

Understanding Reward Hacking in Text-to-Image Reinforcement Learning

Yunqi Hong, Kuei-Chun Kao, Hengguang Zhou, Cho-Jui Hsieh

Abstract

Reinforcement learning (RL) has become a standard approach for post-training large language models and, more recently, for improving image generation models, by optimizing generation quality and human preference alignment with reward signals. However, existing rewards are often imperfect proxies for true human judgment, making models prone to reward hacking and producing unrealistic or low-quality images that nevertheless achieve high reward scores. In this work, we systematically analyze reward hacking behaviors in textto-image (T2I) RL. We investigate how aesthetic/human preference rewards and prompt-image consistency rewards individually contribute to reward hacking. We further show that ensembling multiple rewards can only partially mitigate this issue. Across diverse reward models, we identify a common failure mode: the generation of artifact-prone images. To address this, we propose ArtifactReward, a lightweight adaptive reward model trained on a small curated dataset to penalize generated artifacts. Experiments demonstrate that incorporating our ArtifactReward into existing RL pipelines significantly improves visual realism and reduces reward hacking across multiple T2I RL setups, validating the effectiveness of our lightweight complementary reward serving as a safeguard against reward hacking.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on27

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  4. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
  5. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image GenerationJiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong et al.NeurIPS 2023 · 1,310 citations

Related papers

  1. Reward Sharpness-Aware Fine-Tuning for Diffusion ModelsKwanyoung Kim, Byeongsu SimCVPR 2026 · 1 citation
  2. WARM: On the Benefits of Weight Averaged Reward ModelsAlexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi et al.ICML 2024 · 145 citations
  3. The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image GenerationWeijia Mao, Hao Chen, Zhenheng Yang, Mike Zheng ShouCVPR 2026 · 10 citations
  4. ODIN: Disentangled Reward Mitigates Hacking in RLHFLichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia et al.ICML 2024 · 119 citations
  5. Improving Text-to-Image Generation with Intrinsic Self-Confidence RewardsSeungwook Kim, Minsu ChoCVPR 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines