Lune

ACL Findings2024

ADAM: Dense Retrieval Distillation with Adaptive Dark Examples

Chongyang Tao, Chang Liu, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, Daxin Jiang

Abstract

To improve the performance of the dualencoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works prepare training instances by pairing each query with one positive and a batch of negatives. However, most hard negatives mined by advanced dense retrieval methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query by strengthening negatives and masking positives in the discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher's confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  • Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
  • Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling cites Hard Negative Mixing for Contrastive Learning
  • Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
  • Optimizing Dense Retrieval Model Training with Hard Negatives cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
  • Optimizing Dense Retrieval Model Training with Hard Negatives cites ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on18

  1. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
  2. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
  3. Hard Negative Mixing for Contrastive LearningYannis Kalantidis, Mert Bülent Sariyildiz, Noé Pion, Philippe Weinzaepfel et al.NeurIPS 2020 · 805 citations
  4. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware SamplingSebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin et al.SIGIR 2021 · 297 citations
  5. Optimizing Dense Retrieval Model Training with Hard NegativesJingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo et al.SIGIR 2021 · 242 citations

Related papers

  1. How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?Yuxin Chen, Zongyang Ma, Ziqi Zhang, Zhongang Qi et al.CVPR 2024
  2. Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense RetrievalHouxing Ren, Linjun Shou, Ning Wu, Ming Gong et al.EMNLP 2022 · 6 citations
  3. PROD: Progressive Distillation for Dense RetrievalZhenghao Lin, Yeyun Gong, Xiao Liu, Hang Zhang et al.WWW 2023 · 33 citations
  4. Cooperative Retriever and Ranker in Deep RecommendersXu Huang, Defu Lian, Jin Chen, Zheng Liu et al.WWW 2023 · 17 citations
  5. Towards Robust Ranker for Text RetrievalYucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao et al.ACL Findings 2023 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines