Lune

AAAI2022Top-tier venue

Well-Classified Examples Are Underestimated in Classification with Deep Neural Networks

Guangxiang Zhao, Wenkai Yang, Xuancheng Ren, Lei Li, Yunfang Wu, Xu Sun

Abstract

The conventional wisdom behind learning deep classification models is to focus on bad-classified examples and ignore wellclassified examples that are far from the decision boundary. For instance, when training with cross-entropy loss, examples with higher likelihoods (i.e., well-classified examples) contribute smaller gradients in back-propagation. However, we theoretically show that this common practice hinders representation learning, energy optimization, and margin growth. To counteract this deficiency, we propose to reward well-classified examples with additive bonuses to revive their contribution to the learning process. This counterexample theoretically addresses these three issues. We empirically support this claim by directly verifying the theoretical results or significant performance improvement with our counterexample on diverse tasks, including image classification, graph classification, and machine translation. Furthermore, this paper shows that we can deal with complex scenarios, such as imbalanced classification, OOD detection, and applications under adversarial attacks, because our idea can solve these three issues. Code is available at https://github.com/lancopku/well-classified-examples-are- underestimated.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework cites A Simple Framework for Contrastive Learning of Visual Representations
  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
  • Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin et al.ICML 2022 · 1,058 citations
  2. PolyLoss: A Polynomial Expansion Perspective of Classification Loss FunctionsZhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk et al.ICLR 2022 · 189 citations

Builds on11

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
  3. RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations
  4. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacksFrancesco Croce, Matthias HeinICML 2020 · 2,337 citations
  5. Energy-based Out-of-distribution DetectionWeitang Liu, Xiaoyun Wang, John D. Owens, Yixuan LiNeurIPS 2020 · 2,213 citations

Related papers

  1. Counterexample Contrastive Learning for Spurious Correlation EliminationJinqiang Wang, Rui Hu, Chaoquan Jiang, Rui Hu et al.ACM MM 2022 · 3 citations
  2. M2m: Imbalanced Classification via Major-to-Minor TranslationJaehyung Kim, Jongheon Jeong, Jinwoo ShinCVPR 2020
  3. Improving the Accuracy of Learning Example Weights for Imbalance ClassificationYuqi Liu, Bin Cao, Jing FanICLR 2022 · 10 citations
  4. RCL: Reliable Continual Learning for Unified Failure DetectionFei Zhu, Zhen Cheng, Xu-Yao Zhang, Cheng-Lin Liu et al.CVPR 2024 · 5 citations
  5. Improving Adversarially Robust Few-shot Image Classification with Generalizable RepresentationsJunhao Dong, Yuan Wang, Jianhuang Lai, Xiaohua XieCVPR 2022 · 30 citations
Dusk over the sea between two cliffs drawn in fine vertical lines