EMNLP Findings2023
Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Abstract
Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. Recent multi-modal OOD detection leverages textual information from in-distribution (ID) class names for visual OOD detection, yet it currently neglects the rich contextual information of ID classes. Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descriptive features for each class. Indiscriminately using such knowledge causes catastrophic damage to OOD detection due to LLMs' hallucinations, as is observed by our analysis. In this paper, we propose to apply world knowledge to enhance OOD detection performance through selective generation from LLMs. Specifically, we introduce a consistency-based uncertainty calibration method to estimate the confidence score of each generation. We further extract visual objects from each image to fully capitalize on the aforementioned world knowledge. Extensive experiments demonstrate that our method consistently outperforms the state-of-the-art.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation cites Energy-based Out-of-distribution Detection
- SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation cites Learning Transferable Visual Models From Natural Language Supervision
- Teaching Large Language Models to Self-Debug cites Language Models are Few-Shot Learners
- Teaching Large Language Models to Self-Debug cites Training language models to follow instructions with human feedback
- Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution Detection cites Energy-based Out-of-distribution Detection
- Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution Detection cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
- Open-Set Image Tagging with Multi-Grained Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoning cites Energy-based Out-of-distribution Detection
- ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and Reasoning cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers4
- SeTAR: Out-of-Distribution Detection with Selective Low-Rank ApproximationYixia Li, Boya Xiong, Guanhua Chen, Yun ChenNeurIPS 2024 · 13 citations
- Open-Set Image Tagging with Multi-Grained Text SupervisionXinyu Huang, Yi-Jie Huang, Youcai Zhang, Weiwei Tian et al.ACM MM 2025 · 13 citations
- ANTS: Adaptive Negative Textual Space Shaping for OOD Detection via Test-Time MLLM Understanding and ReasoningWenjie Zhu, Yabin Zhang, Xin Jin, Wenjun Zeng et al.CVPR 2026 · 12 citations
- Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution DetectionReihaneh Zohrabi, Hosein Hasani, Mahdieh Soleymani Baghshah, Anna Rohrbach et al.NeurIPS 2025 · 6 citations
Builds on29
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Energy-based Out-of-distribution DetectionWeitang Liu, Xiaoyun Wang, John D. Owens, Yixuan LiNeurIPS 2020 · 2,213 citations
- Teaching Large Language Models to Self-DebugXinyun Chen, Maxwell Lin, Nathanael Schärli, Denny ZhouICLR 2024 · 1,085 citations
Related papers
- Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language ModelsZhixia He, Chen Zhao, Minglai Shao, Xintao Wu et al.AAAI 2026
- From Out-of-Distribution Detection to Hallucination Detection: A Geometric ViewLitian Liu, Reza Pourreza, Yubing Jian, Yao Qin et al.ICML 2026 · 1 citation
- Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMsZhikai Ding, Shiyu Ni, Keping BiEMNLP Findings 2025
- Mind the Way You Select Negative Texts: Pursuing the Distance Consistency in OOD Detection with VLMsZhikang Xu, Qianqian Xu, Zitai Wang, Cong Hua et al.CVPR 2026 · 1 citation
- Negative Label Guided OOD Detection with Pretrained Vision-Language ModelsXue Jiang, Feng Liu, Zhen Fang, Hong Chen et al.ICLR 2024 · 73 citations
