Lune

EMNLP Findings2023

A Critical Analysis of Document Out-of-Distribution Detection

Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad I. Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, Tong Sun, Ani Nenkova

Abstract

Large-scale pre-training is widely used in recent document understanding tasks. During deployment, one may expect that models should trigger a conservative fallback policy when encountering out-of-distribution (OOD) samples, which highlights the importance of OOD detection. However, most existing OOD detection methods focus on single-modal inputs such as images or texts. While documents are multimodal in nature, it is underexplored if and how multi-modal information in documents can be exploited for OOD detection. In this work, we first provide a systematic and in-depth analysis on OOD detection for document understanding models. We study the effects of model modality, pre-training, and fine-tuning across various types of OOD inputs. In particular, we find that spatial information is critical for document OOD detection. To better exploit spatial information, we propose a spatial-aware adapter, which serves as a parameter-efficient add-on module to adapt transformer-based language models to the document domain. Extensive experiments show that adding the spatial-aware adapter significantly improves the OOD detection performance compared to directly using the language model and achieves superior performance compared to competitive baselines.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Exploring the Limits of Out-of-Distribution Detection cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Out-of-Distribution Detection for Real-World Settings cites Exploring the Limits of Out-of-Distribution Detection
  • On the Importance of Gradients for Detecting Distributional Shifts in the Wild cites Energy-based Out-of-distribution Detection
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on16

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Energy-based Out-of-distribution DetectionWeitang Liu, Xiaoyun Wang, John D. Owens, Yixuan LiNeurIPS 2020 · 2,213 citations
  3. Scaling Out-of-Distribution Detection for Real-World SettingsDan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou et al.ICML 2022 · 653 citations
  4. On the Importance of Gradients for Detecting Distributional Shifts in the WildRui Huang, Andrew Geng, Yixuan LiNeurIPS 2021 · 515 citations
  5. Exploring the Limits of Out-of-Distribution DetectionStanislav Fort, Jie Ren, Balaji LakshminarayananNeurIPS 2021 · 443 citations

Related papers

  1. SelfDoc: Self-Supervised Document Representation LearningPeizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu et al.CVPR 2021
  2. LaTr: Layout-Aware Transformer for Scene-Text VQAAli Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju et al.CVPR 2022 · 82 citations
  3. DocLLM: A Layout-Aware Generative Language Model for Multimodal Document UnderstandingDongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma et al.ACL 2024 · 37 citations
  4. DocFormer: End-to-End Transformer for Document UnderstandingSrikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie et al.ICCV 2021 · 392 citations
  5. UniDoc: Unified Pretraining Framework for Document UnderstandingJiuxiang Gu, Jason Kuen, Vlad I. Morariu, Handong Zhao et al.NeurIPS 2021 · 118 citations
Dusk over the sea between two cliffs drawn in fine vertical lines