Lune

ICLR2023Top-tier venue

Over-Training with Mixup May Hurt Generalization

Zixuan Liu, Ziqiao Wang, Hongyu Guo, Yongyi Mao

Abstract

Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of deep models trained with SGD. In this work, we report a previously unobserved phenomenon in Mixup training: on a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs, giving rise to a U-shaped generalization curve. This behavior is further aggravated when the size of original dataset is reduced. To help understand such a behavior of Mixup, we show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data. Via analyzing a least-square regression problem with a random feature model, we explain why noisy labels may cause the U-shaped curve to occur: Mixup improves generalization through fitting the clean patterns at the early training stage, but as training progresses, Mixup becomes over-fitting to the noise in the synthetic data. Extensive experiments are performed on a variety of benchmark datasets, validating this explanation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Denoising Mixup for Regression cites Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup
  • Denoising Mixup for Regression cites CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
  • Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup cites CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
  • Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors cites CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
  • Supervision Interpolation via LossMix: Generalizing Mixup for Object Detection and Beyond cites Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup
  • Supervision Interpolation via LossMix: Generalizing Mixup for Object Detection and Beyond cites CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
  • Overfitting in adversarially robust deep learning cites Deep Double Descent: Where Bigger Models and More Data Hurt
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion PriorsPaul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin et al.NeurIPS 2023 · 282 citations
  2. Semi-Supervised Graph Imbalanced RegressionGang Liu, Tong Zhao, Eric Inae, Tengfei Luo et al.KDD 2023 · 20 citations
  3. Selective Mixup Helps with Distribution Shifts, But Not (Only) because of MixupDamien Teney, Jindong Wang, Ehsan AbbasnejadICML 2024 · 9 citations
  4. Supervision Interpolation via LossMix: Generalizing Mixup for Object Detection and BeyondThanh Vu, Baochen Sun, Bodi Yuan, Alex Ngai et al.AAAI 2024 · 8 citations
  5. Denoising Mixup for RegressionZhengzhang Hou, Zhanshan Li, Yanbo Liu, Geoff Nitschke et al.AAAI 2026

Builds on16

  1. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable FeaturesSangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh et al.ICCV 2019 · 5,843 citations
  2. Deep Double Descent: Where Bigger Models and More Data HurtPreetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang et al.ICLR 2020 · 1,108 citations
  3. Overfitting in adversarially robust deep learningLeslie Rice, Eric Wong, J. Zico KolterICML 2020 · 935 citations
  4. Early-Learning Regularization Prevents Memorization of Noisy LabelsSheng Liu, Jonathan Niles-Weed, Narges Razavian, Carlos Fernandez-GrandaNeurIPS 2020 · 798 citations
  5. Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal MixupJang-Hyun Kim, Wonho Choo, Hyun Oh SongICML 2020 · 457 citations

Related papers

  1. The Benefits of Mixup for Feature LearningDifan Zou, Yuan Cao, Yuanzhi Li, Quanquan GuICML 2023 · 36 citations
  2. How Does Mixup Help With Robustness and Generalization?Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani et al.ICLR 2021 · 294 citations
  3. GenLabel: Mixup Relabeling using Generative ModelsJy-yong Sohn, Liang Shang, Hongxu Chen, Jaekyun Moon et al.ICML 2022 · 15 citations
  4. Benign Oscillation of Stochastic Gradient Descent with Large Learning RateMiao Lu, Beining Wu, Xiaodong Yang, Difan ZouICLR 2024 · 9 citations
  5. For Better or For Worse? Learning Minimum Variance Features With Label AugmentationMuthu Chidambaram, Rong GeICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines