ACL Findings2026
Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict
Kaiser Sun, Fan Bai, Mark Dredze
Abstract
Large language models (LLMs) draw on both contextual information and parametric memory, yet these sources can conflict.Prior studies have largely examined this issue in contextual question answering, implicitly assuming that tasks should rely on the provided context, leaving unclear how LLMs behave when tasks require different types and degrees of knowledge utilization.We address this gap with a modelagnostic diagnostic framework that holds underlying knowledge constant while introducing controlled conflicts across tasks with varying knowledge demands.Experiments on representative open-weight and proprietary LLMs show that performance degradation under conflict is driven by both task-specific knowledge reliance and conflict plausibility; that strategies such as rationales or context reiteration increase context reliance, helping context-only tasks but harming those requiring parametric knowledge; and that these effects bias model-based evaluation, calling into question the reliability of LLMs as judges.Overall, our findings reveal that context-memory conflict is inherently taskdependent and motivate task-aware approaches to balancing context and memory in LLM deployment and evaluation. 1 4. Validation 3. Task-Annotation 2. Evidence Creation 1. Parametric Knowledge Collection Knowledge Collection TestLM Query the parametric knowledge with test LM.Curate conflict evidence & explanations Curate tasks using: -Knowledge Free -Contextual -Parametric -Combined Validate all data.Subsample for human reviews.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on11
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Towards Understanding Sycophancy in Language ModelsMrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud et al.ICLR 2024 · 762 citations
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
- Humans or LLMs as the Judge? A Study on Judgement BiasGuiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang et al.EMNLP 2024 · 37 citations
- Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting EvidenceHung-Ting Chen, Michael J. Q. Zhang, Eunsol ChoiEMNLP 2022 · 27 citations
Related papers
- Entity-Based Knowledge Conflicts in Question AnsweringShayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh et al.EMNLP 2021 · 3 citations
- Knowledge Conflicts for LLMs: A SurveyRongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang et al.EMNLP 2024 · 38 citations
- Context-faithful Prompting for Large Language ModelsWenxuan Zhou, Sheng Zhang, Hoifung Poon, Muhao ChenEMNLP Findings 2023 · 46 citations
- Conflict-Aware Soft Prompting for Retrieval-Augmented GenerationEunseong Choi, June Park, Hyeri Lee, Jongwuk LeeEMNLP 2025 · 1 citation
- Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge ConflictsJian Xie, Kai Zhang, Jiangjie Chen, Renze Lou et al.ICLR 2024 · 294 citations
