EMNLP Findings2023
Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding
Yu Zhang, Hao Cheng, Zhihong Shen, Xiaodong Liu, Ye-Yi Wang, Jianfeng Gao
Abstract
Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pretrained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme multi-label paper classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques -taskaware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy, where we outperform stateof-the-art scientific pre-trained LMs. Code, datasets, and pre-trained models can be found at https://scimult.github.io/.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts cites Finetuned Language Models are Zero-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery cites Chain-of-Factors Paper-Reviewer Matching
- A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery cites BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers
- A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery cites Training language models to follow instructions with human feedback
- Chain-of-Factors Paper-Reviewer Matching cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- Chain-of-Factors Paper-Reviewer Matching cites Finetuned Language Models are Zero-Shot Learners
- Chain-of-Factors Paper-Reviewer Matching cites Training language models to follow instructions with human feedback
- Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
- Training language models to follow instructions with human feedback cites Multitask Prompted Training Enables Zero-Shot Task Generalization
Cited by top-tier papers7
- A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific DiscoveryYu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang et al.EMNLP 2024 · 28 citations
- HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation PredictionQianyue Hao, Jingyang Fan, Fengli Xu, Jian Yuan et al.NeurIPS 2024 · 23 citations
- Chain-of-Factors Paper-Reviewer MatchingYu Zhang, Yanzhen Shen, SeongKu Kang, Xiusi Chen et al.WWW 2025 · 12 citations
- Weakly Supervised Multi-Label Classification of Full-Text Scientific PapersYu Zhang, Bowen Jin, Xiusi Chen, Yanzhen Shen et al.KDD 2023 · 8 citations
- BMRetriever: Tuning Large Language Models as Better Biomedical Text RetrieversRan Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang et al.EMNLP 2024 · 8 citations
Builds on25
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
- ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
Related papers
- SciLitLLM: How to Adapt LLMs for Scientific Literature UnderstandingSihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi et al.ICLR 2025 · 1 citation
- Fine-Tuning Language Models on Multiple Datasets for Citation Intention ClassificationZeren Shui, Petros Karypis, Daniel S. Karls, Mingjian Wen et al.EMNLP Findings 2024 · 12 citations
- SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific LiteratureDavid Wadden, Kejian Shi, Jacob Morrison, Alan Li et al.EMNLP 2025 · 2 citations
- Structure-Enhanced Protein Instruction Tuning: Towards General-Purpose Protein Understanding with LLMsWei Wu, Chao Wang, Liyi Chen, Mingze Yin et al.KDD 2025 · 1 citation
- Omni-Mol: Multitask Molecular Model for Any-to-any ModalitiesChengxin Hu, Hao Li, Yihe Yuan, Zezheng Song et al.NeurIPS 2025 · 5 citations
