Lune

ACL Findings2026

Characterizing the Robustness of Black-Box LLM Planners Under Perturbed Observations with Adaptive Stress Testing

Neeloy Chakraborty, John Pohovey, Melkior Ornik, Katherine Rose Driggs-Campbell

Abstract

Large language models (LLMs) have recently demonstrated success in decision-making tasks including planning, control, and prediction, but their tendency to hallucinate unsafe and undesired outputs poses risks. This unwanted behavior is further exacerbated in environments where sensors are noisy or unreliable. Characterizing the behavior of LLM planners to varied observations is necessary to proactively avoid failures in safety-critical scenarios. We specifically investigate the response of LLMs along two different perturbation dimensions. Like prior works, one dimension generates semantically similar prompts with varied phrasing by randomizing order of details, modifying access to few-shot examples, etc. Unique to our work, the second dimension simulates access to varied sensors and noise to mimic raw sensor or detection algorithm failures. An initial case study in which perturbations are manually applied show that both dimensions lead LLMs to hallucinate in a multi-agent driving environment. However, manually covering the entire perturbation space for several scenarios is infeasible. As such, we propose a novel method for efficiently searching the space of prompt perturbations using adaptive stress testing (AST) with Monte-Carlo tree search (MCTS). Our AST formulation enables discovery of scenarios, sensor configurations, and prompt phrasing that cause language models to act with high uncertainty or even crash. By generating MCTS prompt perturbation trees across diverse scenarios, we show through extensive experiments that offline analyses can be used to proactively understand potential failures that may arise at runtime. Code is available at https://sites.google.com/illinois.edu/astllm/ .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models cites BARTScore: Evaluating Generated Text as Text Generation
  • BARTScore: Evaluating Generated Text as Text Generation cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on13

  1. PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
  2. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  3. BARTScore: Evaluating Generated Text as Text GenerationWeizhe Yuan, Graham Neubig, Pengfei LiuNeurIPS 2021 · 1,143 citations
  4. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge DistillationJianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo et al.ACL Findings 2024 · 762 citations
  5. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language ModelsPotsawee Manakul, Adian Liusie, Mark J. F. GalesEMNLP 2023 · 331 citations

Related papers

  1. SolverLLM: Leveraging Test-Time Scaling for Optimization Problem via LLM-Guided SearchDong Li, Xujiang Zhao, Linlin Yu, Yanchi Liu et al.NeurIPS 2025 · 15 citations
  2. PlanU: Large Language Model Reasoning through Planning under UncertaintyZiwei Deng, Mian Deng, Chenjing Liang, Zeming Gao et al.NeurIPS 2025 · 4 citations
  3. AdaPlanner: Adaptive Planning from Feedback with Language ModelsHaotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai et al.NeurIPS 2023 · 257 citations
  4. HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language ModelsTrishna Chakraborty, Udita Ghosh, Xiaopan Zhang, Fahim Faisal Niloy et al.EMNLP Findings 2025 · 4 citations
  5. To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic UncertaintyYasin Abbasi-Yadkori, Ilja Kuzborskij, András György, Csaba SzepesváriNeurIPS 2024
Dusk over the sea between two cliffs drawn in fine vertical lines