Lune

KDD2026Top-tier venue

SDE : Scale-Difference Evolution Knowledge Distillation

Hejie Lu

Abstract

Logit-based knowledge distillation (KD) is pivotal for efficient model compression and cross-architecture learning. However, conventional methods typically rely on static, single-scale logit alignment, thereby overlooking the semantic evolution trajectory embedded in cross-scale prediction transitions. To bridge this gap, we propose Scale-Difference Evolution distillation (SDE), formulated in a structure-aware manner. Unlike the traditional prediction-imitation paradigm, SDE explicitly models the difference vectors between multi-scale logits to capture dynamic logical transitions from local features to global semantics. To optimize the distillation signal-to-noise ratio, SDE decouples scale differences into two complementary modules: Category-Dominant Difference (CDD), which isolates evidence fluctuations for core categories via a Top-K attention mask, and Relational Structure Difference (RSD), which preserves the consistency of inter-class topological evolution. Extensive experiments demonstrate that SDE consistently outperforms state-of-the-art methods across CIFAR-100, Tiny-ImageNet-200, CUB-200 and Stanford Cars. Notably, SDE achieves a significant 8.10% accuracy boost on the CUB-200 fine-grained benchmark, highlighting its superior capability in resolving inter-class ambiguities through cross-scale semantic modeling.

Ask Lune

Know where this paper stands in minutes, not an afternoon.

Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

its abstract · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. Knowledge Distillation with Refined LogitsWujie Sun, Defang Chen, Siwei Lyu, Genlang Chen et al.ICCV 2025 · 13 citations
  2. Scale Decoupled DistillationShicai Wei, Chunbo Luo, Yang LuoCVPR 2024 · 32 citations
  3. Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object DetectionLongrong Yang, Xianpan Zhou, Xuewei Li, Liang Qiao et al.ICCV 2023 · 51 citations
  4. Distilling Knowledge from Heterogeneous Architectures for Semantic SegmentationYanglin Huang, Kai Hu, Yuan Zhang, Zhineng Chen et al.AAAI 2025 · 4 citations
  5. Decoupled Knowledge DistillationBorui Zhao, Quan Cui, Renjie Song, Yiyu Qiu et al.CVPR 2022 · 835 citations
Dusk over the sea between two cliffs drawn in fine vertical lines