Beyond Detection: Evaluating Fallacy Awareness of LLMs in Interactive Scenarios

Large Language Models (LLMs) often fail to recognize fallacious reasoning in real-world interactions, despite strong performance on static fallacy detection tasks. We define this ability as fallacy awareness, the capacity to autonomously perceive and resist fallacies in dynamic, pragmatic contexts. To study this, we introduce ISFallacy, a large-scale Chinese benchmark of 50K interactive scenarios spanning six fallacy types, five social interaction settings, diverse role relationships, and personality traits. We further propose FATE, a twostage evaluation framework that assesses fallacy awareness without explicit cues, combining natural dialogue responses and reasoningbased decisions. Experiments on five representative LLMs reveal a sharp contrast between their high accuracy in static fallacy classification and their poor fallacy awareness in active scenarios. Models are particularly prone to overlooking fallacies in emotion-driven or cooperative contexts, where they tend to prioritize social rapport over logical rigor. Deeper analysis uncovers a cognition-behavior gap and fragile internal representations underlying awareness failures. Our work establishes a foundation for evaluating and enhancing the robustness of LLMs against fallacious reasoning in interactive settings.