EMNLP Findings2022
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
Sishuo Chen, Xiaohan Bi, Rundong Gao, Xu Sun
Abstract
Detecting out-of-distribution (OOD) instances is significant for the safe deployment of NLP models. Among recent textual OOD detection works based on pretrained language models (PLMs), distance-based methods have shown superior performance. However, they estimate sample distance scores in the last-layer CLS embedding space and thus do not make full use of linguistic information underlying in PLMs. To address the issue, we propose to boost OOD detection by deriving more holistic sentence embeddings. On the basis of the observations that token averaging and layer combination contribute to improving OOD detection, we propose a simple embedding approach named Avg-Avg, which averages all token representations from each intermediate layer as the sentence embedding and significantly surpasses the state-of-the-art on a comprehensive suite of benchmarks by a 9.33% FAR95 margin. Furthermore, our analysis demonstrates that it indeed helps preserve general linguistic knowledge in fine-tuned PLMs and substantially benefits detecting background shifts. The simple yet effective embedding method can be applied to fine-tuned PLMs with negligible extra costs, providing a free gain in OOD detection. Our code is available at https: //github.com/lancopku/Avg-Avg.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites On the Sentence Embeddings from Pre-trained Language Models
- Text Meets Topology: Rethinking Out-of-distribution Detection in Text-Rich Networks cites Energy-based Out-of-distribution Detection
Cited by top-tier papers2
Builds on8
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Energy-based Out-of-distribution DetectionWeitang Liu, Xiaoyun Wang, John D. Owens, Yixuan LiNeurIPS 2020 · 2,213 citations
- On the Sentence Embeddings from Pre-trained Language ModelsBohan Li, Hao Zhou, Junxian He, Mingxuan Wang et al.EMNLP 2020 · 538 citations
- Exploring the Limits of Out-of-Distribution DetectionStanislav Fort, Jie Ren, Balaji LakshminarayananNeurIPS 2021 · 443 citations
Related papers
- AP-OOD: Attention Pooling for Out-of- Distribution DetectionClaus Hofmann, Christian Huber, Bernhard Lehner, Daniel Klotz et al.ICLR 2026 · 2 citations
- Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain DetectionRheeya Uppaal, Junjie Hu, Yixuan LiACL 2023 · 9 citations
- BARLE: Background-Aware Representation Learning for Background Shift Out-of-Distribution DetectionHanyu Duan, Yi Yang, Ahmed Abbasi, Kar Yan TamEMNLP Findings 2022 · 2 citations
- Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution DetectionIgnacio Meza De La Jara, Cristian Rodriguez Opazo, Damien Teney, Damith Ranasinghe et al.NeurIPS 2025 · 8 citations
- Unsupervised Layer-Wise Score Aggregation for Textual OOD DetectionMaxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie C. K. Cheung et al.AAAI 2024 · 18 citations
