Lune

ACL Findings2023

Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformers

Jaeyoung Kim, Kyuheon Jung, Dongbin Na, Sion Jang, Eunbin Park, Sungchul Choi

Abstract

For real-world language applications, detecting an out-of-distribution (OOD) sample is helpful to alert users or reject such unreliable samples. However, modern over-parameterized language models often produce overconfident predictions for both in-distribution (ID) and OOD samples. In particular, language models suffer from OOD samples with a similar semantic representation to ID samples since these OOD samples lie near the ID manifold. A rejection network can be trained with ID and diverse outlier samples to detect test OOD samples, but explicitly collecting auxiliary OOD datasets brings an additional burden for data collection. In this paper, we propose a simple but effective method called Pseudo Outlier Exposure (POE) that constructs a surrogate OOD dataset by sequentially masking tokens related to ID classes. The surrogate OOD sample introduced by POE shows a similar representation to ID data, which is most effective in training a rejection network. Our method does not require any external OOD data and can be easily implemented within off-the-shelf Transformers. A comprehensive comparison with state-of-theart algorithms demonstrates POE's competitiveness on several text classification benchmarks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection cites Energy-based Out-of-distribution Detection
  • Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection cites A Simple Framework for Contrastive Learning of Visual Representations
  • Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution Detection cites Supervised Contrastive Learning
  • ReAct: Out-of-distribution Detection With Rectified Activations cites Energy-based Out-of-distribution Detection
  • Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient TransformersFiras Gabetni, Giuseppe Curci, Andrea Pilzer, Subhankar Roy et al.ICLR 2026 · 5 citations
  2. Towards More Trustworthy Deep Code Models by Enabling Out-of-Distribution DetectionYanfu Yan, Viet Duong, Huajie Shao, Denys PoshyvanykICSE 2025 · 1 citation

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  3. Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
  4. Energy-based Out-of-distribution DetectionWeitang Liu, Xiaoyun Wang, John D. Owens, Yixuan LiNeurIPS 2020 · 2,213 citations
  5. ReAct: Out-of-distribution Detection With Rectified ActivationsYiyou Sun, Chuan Guo, Yixuan LiNeurIPS 2021 · 733 citations

Related papers

  1. Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution DetectionChentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu et al.ICML 2024 · 34 citations
  2. On the Powerfulness of Textual Outlier Exposure for Visual OoD DetectionSangha Park, Jisoo Mok, Dahuin Jung, Saehyung Lee et al.NeurIPS 2023 · 18 citations
  3. BARLE: Background-Aware Representation Learning for Background Shift Out-of-Distribution DetectionHanyu Duan, Yi Yang, Ahmed Abbasi, Kar Yan TamEMNLP Findings 2022 · 2 citations
  4. Diversified Outlier Exposure for Out-of-Distribution Detection via Informative ExtrapolationJianing Zhu, Yu Geng, Jiangchao Yao, Tongliang Liu et al.NeurIPS 2023 · 54 citations
  5. Key Feature Replacement of In-Distribution Samples for Out-of-Distribution DetectionJaeyoung Kim, Seo Taek Kong, Dongbin Na, Kyu-Hwan JungAAAI 2023 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines