EMNLP Findings2022
Structural Contrastive Representation Learning for Zero-shot Multi-label Text Classification
Tianyi Zhang, Zhaozhuo Xu, Tharun Medini, Anshumali Shrivastava
Abstract
Zero-shot multi-label text classification (ZMTC) is a fundamental task in natural language processing with applications in the cold start problem of recommendation systems. Ideally, one would learn an expressive representation of both input text and label features so that ZMTC is transformed into a nearest neighbor search problem. However, the existing representation learning approaches for ZMTC struggle with accuracy as well as poor training efficiency. Firstly, the input text is structural, consisting of both short title sentences and long content paragraphs. It is challenging to model the correlation between short label descriptions and long structural input documents. Secondly, the enormous label space in ZMTC forces the existing approaches to perform multi-stage learning with label engineering. As a result, the training overhead is significant. In this paper, we address both problems by introducing an end-to-end structural contrastive representation learning approach. We propose a randomized text segmentation (RTS) technique to generate high-quality contrastive pairs. This RTS technique allows us to model title-content correlation. Additionally, we simplify the multi-stage ZMTC learning strategy by avoiding label engineering. Extensive experiments demonstrate that our approach leads to up to 2.33% improvement in precision@1 and 5.94× speedup in training time on publicly available datasets. Our code is available publicly † .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites An Unsupervised Sentence Embedding Method by Mutual Information Maximization
Builds on12
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- MPNet: Masked and Permuted Pre-training for Language UnderstandingKaitao Song, Xu Tan, Tao Qin, Jianfeng Lu et al.NeurIPS 2020 · 1,957 citations
- An Unsupervised Sentence Embedding Method by Mutual Information MaximizationYan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim et al.EMNLP 2020 · 126 citations
Related papers
- Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text ClassificationYu Zhang, Zhihong Shen, Chieh-Han Wu, Boya Xie et al.WWW 2022 · 34 citations
- From Text Segmentation to Enhanced Representation Learning: A Novel Approach to Multi-Label Classification for Long TextsWang Zhang, Xin Wang, Qian Wang, Tao Deng et al.EMNLP Findings 2024 · 3 citations
- Semantic matching for text classification with complex class descriptionsBrian de Silva, Kuan-Wen Huang, Gwang Lee, Karen Hovsepian et al.EMNLP 2023 · 1 citation
- Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text ClassificationSimon Chi Lok U, Jie He, Víctor Gutiérrez-Basulto, Jeff Z. PanEMNLP Findings 2023 · 18 citations
- Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text ClassificationMengyu Li, Yonghao Liu, Fausto Giunchiglia, Ximing Li et al.WWW 2026 · 7 citations
