Lune

ICLR2025Top-tier venue

What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?

Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, Chunhua Shen

Abstract

Extensive pre-training with large data is indispensable for downstream geometry and semantic visual perception tasks. Thanks to large-scale text-to-image (T2I) pretraining, recent works show promising results by simply fine-tuning T2I diffusion models for a few dense perception tasks. However, several crucial design decisions in this process still lack comprehensive justification, encompassing the necessity of the multi-step diffusion mechanism, training strategy, inference ensemble strategy, and fine-tuning data quality. In this work, we conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors. Our key findings are: 1) High-quality fine-tuning data is paramount for both semantic and geometry perception tasks. 2) As a special case of the diffusion scheduler by setting its hyper-parameters, the multi-step generation can be simplified to a one-step fine-tuning paradigm without any loss of performance. 3) Apart from fine-tuning the diffusion model with only latent space supervision, task-specific supervision is beneficial to enhance fine-grained details. These observations culminate in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailored for dense visual perception tasks exploiting diffusion priors. Different from the previous multi-step methods, our paradigm offers a much faster inference speed, and can be seamlessly integrated with customized perception decoders and loss functions for taskspecific supervision, which can be critical for improving the fine-grained details of predictions. Comprehensive experiments on a diverse set of dense visual perceptual tasks, including monocular depth estimation, surface normal estimation, image segmentation, and matting, are performed to demonstrate the remarkable adaptability and effectiveness of our proposed method.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

41 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • NormalCrafter: Learning Temporally Consistent Normals from Video Diffusion Priors cites High-Resolution Image Synthesis with Latent Diffusion Models
  • NormalCrafter: Learning Temporally Consistent Normals from Video Diffusion Priors cites Denoising Diffusion Probabilistic Models
  • Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers cites Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation
  • Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation cites Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers
  • Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformers cites Denoising Diffusion Probabilistic Models
  • Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation cites Denoising Diffusion Probabilistic Models
  • DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks cites LoRA: Low-Rank Adaptation of Large Language Models
  • DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks cites High-Resolution Image Synthesis with Latent Diffusion Models
  • DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks cites Learning Transferable Visual Models From Natural Language Supervision
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers41

  1. Pixel-Perfect Depth with Semantics-Prompted Diffusion TransformersGangwei Xu, Haotong Lin, Hongcheng Luo, Xianqi Wang et al.NeurIPS 2025 · 58 citations
  2. DICEPTION: A Generalist Diffusion Model for Visual Perceptual TasksCanyu Zhao, Yanlong Sun, Mingyu Liu, Huanyi Zheng et al.NeurIPS 2025 · 45 citations
  3. Dens3R: A Foundation Model for 3D Geometry PredictionXianze Fang, Jingnan Gao, Zhe Wang, Zhuo Chen et al.ICLR 2026 · 45 citations
  4. NormalCrafter: Learning Temporally Consistent Normals from Video Diffusion PriorsYanrui Bin, Wenbo Hu, Haoyuan Wang, Xinya Chen et al.ICCV 2025 · 27 citations
  5. Jasmine: Harnessing Diffusion Prior for Self-supervised Depth EstimationJiyuan Wang, Chunyu Lin, Cheng Guan, Lang Nie et al.NeurIPS 2025 · 26 citations

Builds on39

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  4. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  5. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations

Related papers

  1. Edit2Perceive: Image Editing Diffusion Models Are Strong Dense PerceiversYiqing Shi, Yiren Song, Mike Zheng ShouCVPR 2026 · 4 citations
  2. Unleashing Text-to-Image Diffusion Models for Visual PerceptionWenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu et al.ICCV 2023 · 327 citations
  3. Scaling Properties of Diffusion Models For Perceptual TasksRahul Ravishankar, Zeeshan Patel, Jathushan Rajasegaran, Jitendra MalikCVPR 2025
  4. Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual PerceptionZiqi Pang, Xin Xu, Yu-Xiong WangICLR 2025
  5. Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image SynthesisYanzuo Lu, Manlin Zhang, Andy J. Ma, Xiaohua Xie et al.CVPR 2024 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines