Lune

ICML2022Top-tier venue

Combining Diverse Feature Priors

Saachi Jain, Dimitris Tsipras, Aleksander Madry

Abstract

To improve model generalization, model designers often restrict the features that their models use, either implicitly or explicitly. In this work, we explore the design space of leveraging such feature priors by viewing them as distinct perspectives on the data. Specifically, we find that models trained with diverse sets of feature priors have less overlapping failure modes, and can thus be combined more effectively. Moreover, we demonstrate that jointly training such models on additional (unlabeled) data allows them to correct each other's mistakes, which, in turn, leads to better generalization and resilience to spurious correlations. Code available at https://github.com/MadryLab/copriors

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Spurious Feature Diversification Improves Out-of-distribution Generalization cites Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization
  • Do Adversarially Robust ImageNet Models Transfer Better? cites Certified Robustness to Adversarial Examples with Differential Privacy
  • Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Model Ratatouille: Recycling Diverse Models for Out-of-Distribution GeneralizationAlexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord et al.ICML 2023 · 108 citations
  2. Spurious Feature Diversification Improves Out-of-distribution GeneralizationYong Lin, Lu Tan, Yifan Hao, Honam Wong et al.ICLR 2024 · 34 citations
  3. ImageNet-trained CNNs are not biased towards texture: Revisiting feature reliance through controlled suppressionTom Burgert, Oliver Stoll, Paolo Rota, Begüm DemirNeurIPS 2025 · 19 citations
  4. Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot LearningFei Zhou, Peng Wang, Lei Zhang, Zhenghua Chen et al.NeurIPS 2024 · 16 citations
  5. Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time FeedbackTeresa Yeo, Oguzhan Fatih Kar, Zahra Sodagar, Amir ZamirICCV 2023 · 10 citations

Builds on18

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
  3. Certified Robustness to Adversarial Examples with Differential PrivacyMathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu et al.S&P 2019 · 1,022 citations
  4. Confidence Regularized Self-TrainingYang Zou, Zhiding Yu, Xiaofeng Liu, B. V. K. Vijaya Kumar et al.ICCV 2019 · 901 citations
  5. Do Adversarially Robust ImageNet Models Transfer Better?Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor et al.NeurIPS 2020 · 506 citations

Related papers

  1. Don't Judge an Object by Its Context: Learning to Overcome Contextual BiasKrishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee et al.CVPR 2020
  2. Learning from others' mistakes: Avoiding dataset biases without modeling themVictor Sanh, Thomas Wolf, Yonatan Belinkov, Alexander M. RushICLR 2021 · 123 citations
  3. Identifying and Mitigating Spurious Correlation in Multi-Task LearningJunyi Chai, Shenyu Lu, Xiaoqian WangCVPR 2025
  4. The Two Dimensions of Worst-case Training and Their Integrated Effect for Out-of-domain GeneralizationZeyi Huang, Haohan Wang, Dong Huang, Yong Jae Lee et al.CVPR 2022 · 14 citations
  5. Provably Learning Diverse Features in Multi-View Data with Midpoint MixupMuthu Chidambaram, Xiang Wang, Chenwei Wu, Rong GeICML 2023 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines