EMNLP Findings2024
Probing the Capacity of Language Model Agents to Operationalize Disparate Experiential Context Despite Distraction
Sonny George, Chris Sypherd, Dylan Cashman
Abstract
Large language model (LLM) agents show promise in an increasing number of domains. In many proposed applications, it is expected that the agent reasons over accumulated experience presented in an input prompt. We propose the OEDD (Operationalize Experience Despite Distraction) corpus, a human-annotatorvalidated body of scenarios with pre-scripted agent histories where the agent must make a decision based on disparate experiential information in the presence of a distractor. We evaluate three state-of-the-art LLMs (GPT-3.5 Turbo, GPT-4o, and Gemini 1.5 Pro) using a minimal chain-of-thought prompting strategy and observe that when (1) the input context contains over 1,615 tokens of historical interactions, (2) a crucially decision-informing premise is the rightful conclusion over two disparate environment premises, and (3) a trivial, but distracting red herring fact follows, all LLMs perform worse than random choice at selecting the better of two actions. Our code and test corpus are publicly available at: github.com/sonnygeorge/OEDD.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- WebArena: A Realistic Web Environment for Building Autonomous Agents cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- WebArena: A Realistic Web Environment for Building Autonomous Agents cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- AgentBench: Evaluating LLMs as Agents cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- AgentBench: Evaluating LLMs as Agents cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Builds on10
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
- WebArena: A Realistic Web Environment for Building Autonomous AgentsShuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou et al.ICLR 2024 · 1,197 citations
- AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
Related papers
- AgentRefine: Enhancing Agent Generalization through Refinement TuningDayuan Fu, Keqing He, Yejie Wang, Wentao Hong et al.ICLR 2025
- Can large language models explore in-context?Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang et al.NeurIPS 2024 · 95 citations
- Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMsQianqi Yan, Hongquan Li, Shan Jiang, Yang Zhao et al.EMNLP 2025
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree SearchYanbo Wang, Zixiang Xu, Yue Huang, Chujie Gao et al.NeurIPS 2025 · 11 citations
- R³ Prompting: Review, Rephrase and Resolve for Chain-of-Thought Reasoning in Large Language Models under Noisy ContextQingyuan Tian, Hanlun Zhu, Lei Wang, Yang Li et al.EMNLP Findings 2023 · 3 citations
