Lune

ACL Findings2026

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad, Ali Anwar

Abstract

Generative AI can now synthesize strikingly realistic images from text, yet output quality remains highly sensitive to how prompts are phrased. Direct Preference Optimization (DPO) offers a lightweight, off-policy alternative to RL for automatic prompt engineering, but its token-level regularization leaves semantic inconsistency unchecked as prompts that win higher preference scores can still drift away from the user's intended meaning.

We introduce Sem-DPO, a variant of DPO that preserves semantic consistency yet retains its simplicity and efficiency. Sem-DPO adjusts the DPO loss using a weight based on how different the winning prompt is from the original, reducing the impact of training examples that are semantically misaligned. We provide the first analytical bound on semantic drift for preference-tuned prompt generators, showing that Sem-DPO keeps learned prompts within a provably bounded neighborhood of the original text. On three standard text-to-image prompt-optimization benchmarks and three language models, Sem-DPO achieves 8-12% higher CLIP similarity and 5-9% higher human-preference scores (HPSv2.1, PickScore) than DPO, while also outperforming state-ofthe-art prompt optimization baselines as well as several DPO variants. These findings suggest that strong flat baselines augmented with semantic weighting should become the new standard for prompt-optimization studies and lay the groundwork for broader, semantics-aware preference optimization in language models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Directly Denoising Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Directly Denoising Diffusion Models cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on24

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. Directly Denoising Diffusion ModelsDan Zhang, Jingjing Wang, Feng LuoICML 2024 · 11,724 citations
  4. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  5. Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations

Related papers

  1. Curriculum Direct Preference Optimization for Diffusion and Consistency ModelsFlorinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, Nicu Sebe et al.CVPR 2025
  2. DSPO: Direct Score Preference Optimization for Diffusion Model AlignmentHuaisheng Zhu, Teng Xiao, Vasant G. HonavarICLR 2025
  3. Scalable Ranked Preference Optimization for Text-To-Image GenerationShyamgopal Karthik, Huseyin Coskun, Zeynep Akata, Sergey Tulyakov et al.ICCV 2025 · 1 citation
  4. Importance Sampling for Multi-Negative Multimodal Direct Preference OptimizationXintong Li, Chuhan Wang, Junda Wu, Rohan Surana et al.ICLR 2026 · 5 citations
  5. TIPO: Text to Image with Text Pre-sampling for Prompt OptimizationShih-Ying Yeh, YI LI, SangHyun Park, Giyeong Oh et al.ICLR 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines