Lune

CVPR2021Top-tier venue

Student-Teacher Learning From Clean Inputs to Noisy Inputs

Guanzhe Hong, Zhiyuan Mao, Xiaojun Lin, Stanley H. Chan

Abstract

Feature-based student-teacher learning, a training method that encourages the student's hidden features to mimic those of the teacher network, is empirically successful in transferring the knowledge from a pre-trained teacher network to the student network. Furthermore, recent empirical results demonstrate that, the teacher's features can boost the student network's generalization even when the student's input sample is corrupted by noise. However, there is a lack of theoretical insights into why and when this method of transferring knowledge can be successful between such heterogeneous tasks. We analyze this method theoretically using deep linear networks, and experimentally using nonlinear networks. We identify three vital factors to the success of the method: (1) whether the student is trained to zero training loss; (2) how knowledgeable the teacher is on the clean-input problem; (3) how the teacher decomposes its knowledge in its hidden features. Lack of proper control in any of the three factors leads to failure of the student-teacher learning method.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Proto2Proto: Can you recognize the car, the way I do? cites Distilling Image Dehazing With Heterogeneous Task Imitation
  • Proto2Proto: Can you recognize the car, the way I do? cites Similarity-Preserving Knowledge Distillation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Proto2Proto: Can you recognize the car, the way I do?Monish Keswani, Sriranjani Ramakrishnan, Nishant Reddy, Vineeth N. BalasubramanianCVPR 2022 · 20 citations
  2. SputumLocator: Enhancing Airway Clearance with Auscultation-based Sputum LocalizationYanbin Gong, Wentao Xie, Chi Xu, Qian Zhang et al.UbiComp 2025 · 3 citations

Builds on6

  1. Similarity-Preserving Knowledge DistillationFrederick Tung, Greg MoriICCV 2019 · 1,214 citations
  2. A Comprehensive Overhaul of Feature DistillationByeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park et al.ICCV 2019 · 727 citations
  3. Knowledge Distillation from Internal RepresentationsGustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Z. Yao et al.AAAI 2020 · 199 citations
  4. Knowledge Distillation via Route Constrained OptimizationXiao Jin, Baoyun Peng, Yichao Wu, Yu Liu et al.ICCV 2019 · 196 citations
  5. Distilling Image Dehazing With Heterogeneous Task ImitationMing Hong, Yuan Xie, Cuihua Li, Yanyun QuCVPR 2020

Related papers

  1. Learning Student-Friendly Teacher Networks for Knowledge DistillationDae Young Park, Moon-Hyun Cha, Changwook Jeong, Daesin Kim et al.NeurIPS 2021 · 134 citations
  2. Continual Learning in the Teacher-Student Setup: Impact of Task SimilaritySebastian Lee, Sebastian Goldt, Andrew M. SaxeICML 2021 · 98 citations
  3. Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge AmalgamationChengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song et al.ICCV 2019 · 63 citations
  4. Features are fate: a theory of transfer learning in high-dimensional regressionJavan Tahir, Surya Ganguli, Grant M. RotskoffICML 2025
  5. Does Knowledge Distillation Really Work?Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi et al.NeurIPS 2021 · 318 citations
Dusk over the sea between two cliffs drawn in fine vertical lines