Tree-Like Decision Distillation
Jie Song, Haofei Zhang, Xinchao Wang, Mengqi Xue, Ying Chen, Li Sun, Dacheng Tao, Mingli Song
Abstract
Knowledge distillation pursues a diminutive yet wellbehaved student network by harnessing the knowledge learned by a cumbersome teacher model. Prior methods achieve this by making the student imitate shallow behaviors, such as soft targets, features, or attention, of the teacher. In this paper, we argue that what really matters for distillation is the intrinsic problem-solving process captured by the teacher. By dissecting the decision process in a layer-wise manner, we found that the decision-making procedure in the teacher model is conducted in a coarse-to-fine manner, where coarse-grained discrimination (e.g., animal vs vehicle) is attained in early layers, and fine-grained discrimination (e.g., dog vs cat, car vs truck) in latter layers. Motivated by this observation, we propose a new distillation method, dubbed as Tree-like Decision Distillation (TDD), to endow the student with the same problem-solving mechanism as that of the teacher. Extensive experiments demonstrated that TDD yields competitive performance compared to state of the arts. More importantly, it enjoys better interpretability due to its interpretable decision distillation instead of dark knowledge distillation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Contrastive Representation Distillation cites Similarity-Preserving Knowledge Distillation
- Contrastive Representation Distillation cites Correlation Congruence for Knowledge Distillation
- Proto2Proto: Can you recognize the car, the way I do? cites Contrastive Representation Distillation
- Proto2Proto: Can you recognize the car, the way I do? cites Similarity-Preserving Knowledge Distillation
- Proto2Proto: Can you recognize the car, the way I do? cites Correlation Congruence for Knowledge Distillation
Cited by top-tier papers5
- Joint Video Summarization and Moment Localization by Cross-Task Sample TransferHao Jiang, Yadong MuCVPR 2022 · 45 citations
- Proto2Proto: Can you recognize the car, the way I do?Monish Keswani, Sriranjani Ramakrishnan, Nishant Reddy, Vineeth N. BalasubramanianCVPR 2022 · 20 citations
- Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-trainingHaofei Zhang, Jiarui Duan, Mengqi Xue, Jie Song et al.CVPR 2022 · 19 citations
- Editable XAI: Toward Bidirectional Human-AI Alignment with Co-Editable Explanations of Interpretable AttributesHaoyang Chen, Jingwen Bai, Fang Tian, Brian Y. LimCHI 2026 · 2 citations
- Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge DistillationTianli Zhang, Mengqi Xue, Jiangtao Zhang, Haofei Zhang et al.CVPR 2023
Builds on10
- Improved Knowledge Distillation via Teacher AssistantSeyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine et al.AAAI 2020 · 1,361 citations
- Contrastive Representation DistillationYonglong Tian, Dilip Krishnan, Phillip IsolaICLR 2020 · 1,305 citations
- Similarity-Preserving Knowledge DistillationFrederick Tung, Greg MoriICCV 2019 · 1,214 citations
- Correlation Congruence for Knowledge DistillationBaoyun Peng, Xiao Jin, Dongsheng Li, Shunfeng Zhou et al.ICCV 2019 · 625 citations
- Factorizable Graph Convolutional NetworksYiding Yang, Zunlei Feng, Mingli Song, Xinchao WangNeurIPS 2020 · 175 citations
Related papers
- On the Impact of Knowledge Distillation for Model InterpretabilityHyeongrok Han, Siwon Kim, Hyun-Soo Choi, Sungroh YoonICML 2023 · 13 citations
- Learning Student-Friendly Teacher Networks for Knowledge DistillationDae Young Park, Moon-Hyun Cha, Changwook Jeong, Daesin Kim et al.NeurIPS 2021 · 134 citations
- AD-KD: Attribution-Driven Knowledge Distillation for Language Model CompressionSiyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang et al.ACL 2023 · 10 citations
- Heterogeneous Knowledge Distillation Using Information Flow ModelingNikolaos Passalis, Maria Tzelepi, Anastasios TefasCVPR 2020
- Beyond Logits: Aligning Feature Dynamics for Effective Knowledge DistillationGuoqiang Gong, Jiaxing Wang, Jin Xu, Deping Xiang et al.ACL 2025
