Lune

CVPR2023Top-tier venue

Improving Visual Representation Learning Through Perceptual Understanding

Samyakh Tukra, Frederick Hoffman, Ken Chatfield

Abstract

We present an extension to masked autoencoders (MAE) which improves on the representations learnt by the model by explicitly encouraging the learning of higher scene-level features. We do this by: (i) the introduction of a perceptual similarity term between generated and real images (ii) incorporating several techniques from the adversarial training literature including multi-scale training and adaptive discriminator augmentation. The combination of these results in not only better pixel reconstruction but also representations which appear to capture better higher-level details within images. More consequentially, we show how our method, Perceptual MAE, leads to better performance when used for downstream tasks outperforming previous methods. We achieve 78.1% top-1 accuracy linear probing on ImageNet-1K and up to 88.1% when fine-tuning, with similar results for other downstream tasks, all without use of additional pre-trained models or data.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Emerging Properties in Self-Supervised Vision Transformers cites A Simple Framework for Contrastive Learning of Visual Representations
  • Emerging Properties in Self-Supervised Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites A Simple Framework for Contrastive Learning of Visual Representations
  • Diffusion Models Beat GANs on Image Synthesis cites Zero-Shot Text-to-Image Generation
  • ActiveDC: Distribution Calibration for Active Finetuning cites Emerging Properties in Self-Supervised Vision Transformers
  • ActiveDC: Distribution Calibration for Active Finetuning cites A Simple Framework for Contrastive Learning of Visual Representations
  • ActiveDC: Distribution Calibration for Active Finetuning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. ActiveDC: Distribution Calibration for Active FinetuningWenshuai Xu, Zhenghui Hu, Yu Lu, Jinzhou Meng et al.CVPR 2024 · 5 citations
  2. Harnessing Global-Local Collaborative Adversarial Perturbation for Anti-CustomizationLong Xu, Jiakai Wang, Haojie Hao, Haotong Qin et al.CVPR 2025

Builds on14

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  4. Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou et al.ICCV 2021 · 8,921 citations
  5. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations

Related papers

  1. Masked Auto-Encoders Meet Generative Adversarial Networks and BeyondZhengcong Fei, Mingyuan Fan, Li Zhu, Junshi Huang et al.CVPR 2023
  2. 3D Feature Prediction for Masked-AutoEncoder-Based Point Cloud PretrainingSiming Yan, Yuqi Yang, Yu-Xiao Guo, Hao Pan et al.ICLR 2024 · 21 citations
  3. Mixed Autoencoder for Self-Supervised Visual Representation LearningKai Chen, Zhili Liu, Lanqing Hong, Hang Xu et al.CVPR 2023
  4. How Mask Matters: Towards Theoretical Understandings of Masked AutoencodersQi Zhang, Yifei Wang, Yisen WangNeurIPS 2022 · 119 citations
  5. Self-Guided Masked AutoencoderJeongwoo Shin, Inseo Lee, Junho Lee, Joonseok LeeNeurIPS 2024 · 18 citations
Dusk over the sea between two cliffs drawn in fine vertical lines