Truth or Sophistry? LoFa: A Benchmark for LLM Robustness Against Logical Fallacies

While Large Language Models (LLMs) exhibit strong semantic capabilities, their resilience to manipulative linguistic patterns like logical fallacies remains an underexplored area. Prior work has focused on the ability of LLMs to identify or classify fallacies, but their robustness against these fallacies in persuasive contexts remains largely unexplored. To address this gap, we introduce LoFa (Logical Fallacy), a comprehensive benchmark to evaluate LLM robustness against fallacies. We first construct the LoFa dataset via a multi-agent pipeline, pairing factual questions with fallacious arguments. Then, we develop a multi-round debate framework to assess model resilience under sustained attacks. Furthermore, to disentangle robustness from a model's inherent knowledge limitations, we propose a new metric, LFR@k (Logical Fallacy Resistance), to quantify performance. Our experiments reveal that different LLMs exhibit varied robustness to distinct types of fallacies, highlighting unique vulnerability profiles across models. * Equal Contribution. † Corresponding Author. The dataset and evaluation code are available at https: //github.com/xdshen-ai/LoFa . ...The ground beneath your feet was likely covered in sand, right? Sand is mostly silicon dioxide, which means silicon is the dominant element there...