Lune

AAAI2021Top-tier venue

Collaborative Group Learning

Shaoxiong Feng, Hongshen Chen, Xuancheng Ren, Zhuoye Ding, Kan Li, Xu Sun

Abstract

Collaborative learning has successfully applied knowledge transfer to guide a pool of small student networks towards robust local minima. However, previous approaches typically struggle with drastically aggravated student homogenization when the number of students rises. In this paper, we propose Collaborative Group Learning, an efficient framework that aims to diversify the feature representation and conduct an effective regularization. Intuitively, similar to the human group study mechanism, we induce students to learn and exchange different parts of course knowledge as collaborative groups. First, each student is established by randomly routing on a modular neural network, which facilitates flexible knowledge communication between students due to random levels of representation sharing and branching. Second, to resist the student homogenization, students first compose diverse feature sets by exploiting the inductive bias from sub-sets of training data, and then aggregate and distill different complementary knowledge by imitating a random sub-group of students at each time step. Overall, the above mechanisms are beneficial for maximizing the student population to further improve the model generalization without sacrificing computational efficiency. Empirical evaluations on both image and text tasks indicate that our method significantly outperforms various state-of-the-art collaborative approaches whilst enhancing computational efficiency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing cites Online Knowledge Distillation with Diverse Peers
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight SharingChen Liang, Pengcheng He, Yelong Shen, Weizhu Chen et al.ACL 2022 · 6 citations

Builds on2

  1. Online Knowledge Distillation with Diverse PeersDefang Chen, Jian-Ping Mei, Can Wang, Yan Feng et al.AAAI 2020 · 354 citations
  2. Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training DataFelipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley et al.ICML 2020 · 180 citations

Related papers

  1. Online Knowledge Distillation via Collaborative LearningQiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu et al.CVPR 2020
  2. Diverse Client Selection for Federated Learning via Submodular MaximizationRavikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat et al.ICLR 2022 · 140 citations
  3. Heterogeneous-Branch Collaborative Learning for Dialogue GenerationYiwei Li, Shaoxiong Feng, Bin Sun, Kan LiAAAI 2023 · 4 citations
  4. Multi-View Feature Representation for Dialogue Generation with Bidirectional DistillationShaoxiong Feng, Xuancheng Ren, Kan Li, Xu SunAAAI 2021 · 13 citations
  5. ORC: Network Group-based Knowledge Distillation using Online Role ChangeJunyong Choi, Hyeon Cho, Seokhwa Cheung, Wonjun HwangICCV 2023 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines