Lune

EMNLP Findings2025

PROOD: A Simple LLM Out-of-Distribution Guardrail Leveraging Response Semantics

Joshua Tint

Abstract

Out-of-distribution (OOD) detection is a key safeguard for large language models, especially when they're deployed in real-world applications. However, existing OOD methods often struggle with prompts that are deliberately obfuscated, context-dependent, or superficially benign-making it hard to distinguish between harmless queries and adversarial or dangerous ones. These methods typically assess prompts in isolation, missing important semantic cues from the model's response. We introduce PROOD, prompt-response OOD detection, a framework that jointly analyzes LLM prompts and their corresponding outputs to improve semantic understanding. PROOD supports zeroshot multiclass detection using synthetic data generation and it offers a tunable probabilistic classification output. We validate PROOD on three challenging benchmarks-TrustLLM, OR-Bench, and AdvBench-where consistently outperforms prior OOD techniques, improving F1 scores by up to 6.3 points, from 0.871 to 0.934. Our results show that incorporating model responses enables more accurate, context-aware OOD detection in complex and adversarial prompt environments.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on8

  1. Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu et al.ACL 2023 · 540 citations
  2. MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to UseYue Huang, Jiawen Shi, Yuan Li, Chenrui Fan et al.ICLR 2024 · 188 citations
  3. Do Large Language Models Know What They Don't Know?Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu et al.ACL Findings 2023 · 75 citations
  4. Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain DetectionRheeya Uppaal, Junjie Hu, Yixuan LiACL 2023 · 9 citations
  5. Efficient Out-of-Domain Detection for Sequence to Sequence ModelsArtem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov et al.ACL Findings 2023 · 6 citations

Related papers

  1. IPAD: Inverse Prompt for AI Detection - A Robust and Interpretable LLM-Generated Text DetectorZheng Chen, Yushi Feng, Jisheng Dang, Changyang He et al.NeurIPS 2025 · 2 citations
  2. Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution DetectionChentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu et al.ICML 2024 · 34 citations
  3. From Out-of-Distribution Detection to Hallucination Detection: A Geometric ViewLitian Liu, Reza Pourreza, Yubing Jian, Yao Qin et al.ICML 2026 · 1 citation
  4. Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language ModelsZhixia He, Chen Zhao, Minglai Shao, Xintao Wu et al.AAAI 2026
  5. Exploring Large Language Models for Multi-Modal Out-of-Distribution DetectionYi Dai, Hao Lang, Kaisheng Zeng, Fei Huang et al.EMNLP Findings 2023 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines