EMNLP Findings2023
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings
Abstract
Existing model pretraining methods only consider local information. For example, in the popular token masking strategy, the words closer to the masked token are more important for prediction than words far away. This results in pretrained models that generate high-quality sentence embeddings, but low-quality embeddings for large documents. We propose a new pretraining method called DOCSPLIT which forces models to consider the entire global context of a large document. Our method uses a contrastive loss where the positive examples are randomly sampled sections of the input document, and negative examples are randomly sampled sections of unrelated documents. Like previous pretraining methods, DOCSPLIT is fully unsupervised, easy to implement, and can be used to pretrain any model architecture. Our experiments show that DOCSPLIT outperforms other pretraining methods for document classification, few shot learning, and document retrieval tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Long Range Arena : A Benchmark for Efficient Transformers cites Big Bird: Transformers for Longer Sequences
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
Builds on7
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Long Range Arena : A Benchmark for Efficient TransformersYi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen et al.ICLR 2021 · 881 citations
- One Embedder, Any Task: Instruction-Finetuned Text EmbeddingsHongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang et al.ACL Findings 2023 · 134 citations
Related papers
- UniDoc: Unified Pretraining Framework for Document UnderstandingJiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao et al.NeurIPS 2021 · 118 citations
- SelfDoc: Self-Supervised Document Representation LearningPeizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu et al.CVPR 2021
- Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation ModelsLorenzo Lupo, Marco Dinarelli, Laurent BesacierACL 2022 · 21 citations
- Pre-training for Abstractive Document Summarization by Reinstating Source TextYanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei et al.EMNLP 2020 · 42 citations
- CSP: Code-Switching Pre-training for Neural Machine TranslationZhen Yang, Bojie Hu, Ambyera Han, Shen Huang et al.EMNLP 2020 · 62 citations
