EMNLP Findings2021
Contrastive Document Representation Learning with Graph Attention Networks
Peng Xu, Xinchi Chen, Xiaofei Ma, Zhiheng Huang, Bing Xiang
Abstract
Recent progress in pretrained Transformerbased language models has shown great success in learning contextual representation of text. However, due to the quadratic selfattention complexity, most of the pretrained Transformers models can only handle relatively short text. It is still a challenge when it comes to modeling very long documents. In this work, we propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. This graph attention network allows us to leverage the high-level semantic structure of the document. In addition, based on our graph document model, we design a simple contrastive learning strategy to pretrain our models on a large amount of unlabeled corpus. Empirically, we demonstrate the effectiveness of our approaches in document classification and document retrieval tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Coreference Graph Guidance for Mind-Map Generation cites A Simple Framework for Contrastive Learning of Visual Representations
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining cites Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining cites A Simple Framework for Contrastive Learning of Visual Representations
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites Dense Passage Retrieval for Open-Domain Question Answering
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval cites A Simple Framework for Contrastive Learning of Visual Representations
- Recurrent Memory Transformer cites Big Bird: Transformers for Longer Sequences
Cited by top-tier papers3
- Recurrent Memory TransformerAydar Bulatov, Yuri Kuratov, Mikhail BurtsevNeurIPS 2022 · 252 citations
- SciRepEval: A Multi-Format Benchmark for Scientific Document RepresentationsAmanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey et al.EMNLP 2023 · 45 citations
- Coreference Graph Guidance for Mind-Map GenerationZhuowei Zhang, Mengting Hu, Yinhao Bai, Zhen ZhangAAAI 2024 · 2 citations
Builds on8
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
- COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingYu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary et al.NeurIPS 2021 · 231 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
Related papers
- Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document ClassificationSudipta Singha Roy, Xindi Wang, Robert E. Mercer, Frank RudziczEMNLP Findings 2024 · 1 citation
- Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks?Margarita Bugueño, Gerard de MeloEMNLP Findings 2023 · 8 citations
- NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long DocumentsTamara Czinczoll, Christoph Hönes, Maximilian Schall, Gerard de MeloACL 2024 · 4 citations
- Span Graph Transformer for Document-Level Named Entity RecognitionHongli Mao, Xian-Ling Mao, Hanlin Tang, Yuming Shang et al.AAAI 2024 · 3 citations
- A Graph-based Relevance Matching Model for Ad-hoc RetrievalYufeng Zhang, Jinghao Zhang, Zeyu Cui, Shu Wu et al.AAAI 2021 · 26 citations
