Lune

ICLR2025Top-tier venue

Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception

Ziqi Pang, Xin Xu, Yu-Xiong Wang

Abstract

With success in image generation, generative diffusion models are increasingly adopted for discriminative scenarios because generating pixels is a unified and natural perception interface. Although directly re-purposing their generative denoising process has established promising progress in specialist (e.g., depth estimation) and generalist models, the inherent gaps between a generative process and discriminative objectives are rarely investigated. For instance, generative models can tolerate deviations at intermediate sampling steps as long as the final distribution is reasonable, while discriminative tasks with rigorous ground truth for evaluation are sensitive to such errors. Without mitigating such gaps, diffusion for perception still struggles on tasks represented by multi-modal understanding (e.g., referring image segmentation). Motivated by these challenges, we analyze and improve the alignment between the generative diffusion process and perception objectives centering around the key observation: how perception quality evolves with the denoising process. (1) Notably, earlier denoising steps contribute more than later steps, necessitating a tailored learning objective for training: loss functions should reflect varied contributions of timesteps for each perception task.

(2) Perception quality drops unexpectedly at later denoising steps, revealing the sensitiveness of perception to training-denoising distribution shift. We introduce diffusion-tailored data augmentation to simulate such drift in the training data.

(3) We suggest a novel perspective to the long-standing question: why should a generative process be useful for discriminative tasks -interactivity. The denoising process can be leveraged as a controllable user interface adapting to users' correctional prompts and conducting multi-round interaction in an agentic workflow. Collectively, our insights enhance multiple generative diffusion-based perception models without architectural changes: state-of-the-art diffusion-based depth estimator, previously underplayed referring image segmentation models, and perception generalists. Our code is available at https://github.com/ziqipang/ADDP.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
  • FlowDIS: Language-Guided Dichotomous Image Segmentation with Flow Matching cites High-Resolution Image Synthesis with Latent Diffusion Models
  • FlowDIS: Language-Guided Dichotomous Image Segmentation with Flow Matching cites Denoising Diffusion Probabilistic Models
  • Visual Instruction Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Implicit Models
  • Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. FlowDIS: Language-Guided Dichotomous Image Segmentation with Flow MatchingAndranik Sargsyan, Shant NavasardyanCVPR 2026

Builds on43

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  3. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  4. Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
  5. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations

Related papers

  1. Text-Image Alignment for Diffusion-Based PerceptionNeehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães et al.CVPR 2024
  2. Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion ModelsShuhong Zheng, Zhipeng Bao, Ruoyu Zhao, Martial Hebert et al.ICLR 2025
  3. What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan et al.ICLR 2025
  4. Reconciling Visual Perception and Generation in Diffusion ModelsLiulei Li, Yi Yang, Wenguan WangICLR 2026
  5. Unleashing Text-to-Image Diffusion Models for Visual PerceptionWenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu et al.ICCV 2023 · 327 citations
Dusk over the sea between two cliffs drawn in fine vertical lines