Lune

CVPR2024Top-tier venue

Learning to Rank Patches for Unbiased Image Redundancy Reduction

Yang Luo, Zhineng Chen, Peng Zhou, Zuxuan Wu, Xieping Gao, Yu-Gang Jiang

Abstract

Images suffer from heavy spatial redundancy because pixels in neighboring regions are spatially correlated. Existing approaches strive to overcome this limitation by reducing less meaningful image regions. However, current leading methods rely on supervisory signals. They may compel models to preserve content that aligns with labeled categories and discard content belonging to unlabeled categories. This categorical inductive bias makes these methods less effective in real-world scenarios. To address this issue, we propose a self-supervised framework for image redundancy reduction called Learning to Rank Patches (LTRP). We observe that image reconstruction of masked image modeling models is sensitive to the removal of visible patches when the masking ratio is high (e.g., 90%). Building upon it, we implement LTRP via two steps: inferring the semantic density score of each patch by quantifying variation between reconstructions with and without this patch, and learning to rank the patches with the pseudo score. The entire process is self-supervised, thus getting out of the dilemma of categorical inductive bias. We design extensive experiments on different datasets and tasks. The results demonstrate that LTRP outperforms both supervised and other self-supervised methods due to the fair assessment of image content. Code is available at https://github.com/irsLu/ltrp .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Emerging Properties in Self-Supervised Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision cites DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
  • LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision cites SimMIM: a Simple Framework for Masked Image Modeling
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • SimMIM: a Simple Framework for Masked Image Modeling cites Emerging Properties in Self-Supervised Vision Transformers
  • SimMIM: a Simple Framework for Masked Image Modeling cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • SimMIM: a Simple Framework for Masked Image Modeling cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • Sparse Imagination for Efficient Visual World Model Planning cites Emerging Properties in Self-Supervised Vision Transformers
  • Sparse Imagination for Efficient Visual World Model Planning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Sparse Imagination for Efficient Visual World Model Planning cites DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Sparse Imagination for Efficient Visual World Model PlanningJunha Chun, Youngjoon Jeong, Taesup KimICLR 2026 · 8 citations
  2. LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-SupervisionAnthony Fuller, Yousef Yassin, Junfeng Wen, Tarek Ibrahim et al.NeurIPS 2025 · 7 citations

Builds on20

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou et al.ICCV 2021 · 8,921 citations
  4. DynamicViT: Efficient Vision Transformers with Dynamic Token SparsificationYongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu et al.NeurIPS 2021 · 1,343 citations
  5. SimMIM: a Simple Framework for Masked Image ModelingZhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin et al.CVPR 2022 · 1,129 citations

Related papers

  1. Global Patch-wise Attention is Masterful Facilitator for Masked Image ModelingGongli Xi, Ye Tian, Mengyu Yang, Lanshan Zhang et al.ACM MM 2024 · 1 citation
  2. Progressively Compressed Auto-Encoder for Self-supervised Representation LearningJin Li, Yaoming Wang, Xiaopeng Zhang, Yabo Chen et al.ICLR 2023
  3. Adaptive Multi-Modality Prompt LearningZongqian Wu, Yujing Liu, Mengmeng Zhan, Ping Hu et al.ACM MM 2024 · 1 citation
  4. Self-Supervised Debiasing Using Low Rank RegularizationGeon Yeong Park, Chanyong Jung, Sangmin Lee, Jong Chul Ye et al.CVPR 2024 · 2 citations
  5. Estimating Low-Rank Region Likelihood MapsGabriela Csurka, Zoltan Kato, Andor Juhasz, Martin HumenbergerCVPR 2020
Dusk over the sea between two cliffs drawn in fine vertical lines