Lune

ACL Findings2025

Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactions

Yubo Li, Yidi Miao, Xueying Ding, Ramayya Krishnan, Rema Padman

Abstract

Large Language Models (LLMs) have shown remarkable capabilities across various tasks, but their deployment in high-stake domains requires consistent and coherent behavior across multiple rounds of user interaction. This paper introduces a comprehensive framework for evaluating and improving LLM response consistency, making three key contributions 1 . First, we introduce Position-Weighted Consistency (PWC), a metric designed to capture both the importance of early-stage stability and recovery patterns in multi-turn interactions. Second, we present MT-Consistency, a carefully curated benchmark dataset spanning diverse domains and difficulty levels, specifically designed to evaluate LLM consistency under various challenging follow-up scenarios. Third, we introduce Confidence-Aware Response Generation (CARG), a framework that significantly improves response stability by explicitly integrating internal model confidence scores during the generation process. Experimental results demonstrate that CARG significantly improves response stability without sacrificing accuracy, offering a practical path toward more dependable LLM behavior in critical, real-world deployments.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting cites Large Language Models are Zero-Shot Reasoners
  • Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks cites Measuring Massive Multitask Language Understanding
  • Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks cites Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  • Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Interaction Context Often Increases Sycophancy in LLMsShomik Jain, Charlotte Park, Matt Viana, Ashia Wilson et al.CHI 2026 · 12 citations
  2. Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial AttacksYubo Li, Ramayya Krishnan, Rema PadmanICLR 2026 · 6 citations
  3. Evaluating Temporal Consistency in Multi-Turn Language ModelsYash Kumar Atri, Steven L. Johnson, Thomas HartvigsenACL 2026 · 1 citation
  4. Not All Answers Are Contextually Persuadable: Inference Dynamics in Large Language Models under Contextual InfluenceZongye Hu, Weiqing Luo, Yanjie Fu, Yu Gan et al.ICML 2026

Builds on18

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations

Related papers

  1. Consistently Simulating Human Personas with Multi-Turn Reinforcement LearningMarwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff et al.NeurIPS 2025 · 51 citations
  2. CER: Confidence Enhanced Reasoning in LLMsAli Razghandi, Seyed Mohammad Hadi Hosseini, Mahdieh Soleymani BaghshahACL 2025 · 11 citations
  3. Calibrating Long-form Generations From Large Language ModelsYukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan et al.EMNLP Findings 2024 · 19 citations
  4. Ask Again, Then Fail: Large Language Models' Vacillations in JudgmentQiming Xie, Zengzhi Wang, Yi Feng, Rui XiaACL 2024
  5. RefuteBench: Evaluating Refuting Instruction-Following for Large Language ModelsJianhao Yan, Yun Luo, Yue ZhangACL Findings 2024 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines