Lune

EMNLP Findings2023

DistillCSE: Distilled Contrastive Learning for Sentence Embeddings

Jiahao Xu, Wei Shao, Lihui Chen, Lemao Liu

Abstract

This paper proposes the DistillCSE framework, which performs contrastive learning under the self-training paradigm with knowledge distillation. The potential advantage of DistillCSE is its self-enhancing feature: using a base model to provide additional supervision signals, a stronger model may be learned through knowledge distillation. However, the vanilla Dis-tillCSE through the standard implementation of knowledge distillation only achieves marginal improvements due to severe overfitting. The further quantitative analyses demonstrate the reason that the standard knowledge distillation exhibits a relatively large variance of the teacher model's logits due to the essence of contrastive learning. To mitigate the issue induced by high variance, this paper accordingly proposed two simple yet effective solutions for knowledge distillation: a Group-P shuffling strategy as an implicit regularization and the averaging logits from multiple teacher components. Experiments on standard benchmarks demonstrate that the proposed DistillCSE outperforms many strong baseline methods and yields a new state-of-the-art performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
  • TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation cites On the Sentence Embeddings from Pre-trained Language Models
  • TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation cites ESE: Espresso Sentence Embeddings
  • TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding Distillation cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives cites On the Sentence Embeddings from Pre-trained Language Models
  • SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two Perspectives cites ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
  • ESE: Espresso Sentence Embeddings cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
  • SimCSE: Simple Contrastive Learning of Sentence Embeddings cites On the Sentence Embeddings from Pre-trained Language Models
  • SimCSE: Simple Contrastive Learning of Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
  • Static Word Embeddings for Sentence Semantic Representation cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. SimCSE++: Improving Contrastive Learning for Sentence Embeddings from Two PerspectivesJiahao Xu, Wei Shao, Lihui Chen, Lemao LiuEMNLP 2023 · 7 citations
  2. Static Word Embeddings for Sentence Semantic RepresentationTakashi Wada, Yuki Hirakawa, Ryotaro Shimizu, Takahiro Kawashima et al.EMNLP 2025 · 1 citation
  3. TALAS: Teacher-Anchored Layer Alignment with Adaptive Sharpness-Aware Minimization for Embedding DistillationQuoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi, Linh Ngo Van et al.ACL 2026
  4. ESE: Espresso Sentence EmbeddingsXianming Li, Zongxi Li, Jing Li, Haoran Xie et al.ICLR 2025

Builds on22

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  3. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  4. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than GeneratorsKevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. ManningICLR 2020 · 541 citations
  5. On the Sentence Embeddings from Pre-trained Language ModelsBohan Li, Hao Zhou, Junxian He, Mingxuan Wang et al.EMNLP 2020 · 538 citations

Related papers

  1. Prototypical Contrastive Predictive CodingKyungmin LeeICLR 2022 · 9 citations
  2. SEED: Self-supervised Distillation For Visual RepresentationZhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang et al.ICLR 2021 · 213 citations
  3. Boosting Contrastive Learning with Relation Knowledge DistillationKai Zheng, Yuanjiang Wang, Ye YuanAAAI 2022 · 15 citations
  4. Complementary Relation Contrastive DistillationJinguo Zhu, Shixiang Tang, Dapeng Chen, Shijie Yu et al.CVPR 2021
  5. Pay Attention to Your Positive Pairs: Positive Pair Aware Contrastive Knowledge DistillationZhipeng Yu, Qianqian Xu, Yangbangyan Jiang, Haoyu Qin et al.ACM MM 2022 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines