ACL Findings2026
SEC-FinTables: Evaluating Large Language Models for Detecting Logical Inconsistencies on Tabular Data
Shuyan Ke, Qiong Wu, Hui Li, Liujuan Cao
Abstract
Large language models (LLMs) are increasingly deployed in high-stakes domains reliant on tabular data (e.g., financial reporting), where undetected logical inconsistencies such as mismatched totals and components can lead to critical errors. Yet, the ability of LLMs to identify such inconsistencies remains poorly understood, hindered by the absence of standardized evaluation frameworks and cell-level annotated datasets. To bridge this gap, we propose a comprehensive benchmark SEC-FINTABLES comprising 103,395 real-world and error-injected table instances, alongside a novel evaluation protocol that decomposes inconsistency detection into granular sub-tasks. Through evaluating both proprietary and open-source LLMs on SEC-FINTABLES, we find that contemporary LLMs exhibit only partial competence in detecting logical inconsistencies. Our study reveals key limitations and improvement opportunities for LLMs. We believe SEC-FINTABLES and our evaluation protocol can serve as a practical resource for advancing reliable inconsistency detection of LLMs in tabular reasoning. We release SEC-FINTABLES at https: //github.com/XIEFOX/SEC-Fintables.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data cites TabFact: A Large-scale Dataset for Table-based Fact Verification
- TGRNet: A Table Graph Reconstruction Network for Table Structure Recognition cites TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data cites TabFact: A Large-scale Dataset for Table-based Fact Verification
- Open Question Answering over Tables and Text cites TabFact: A Large-scale Dataset for Table-based Fact Verification
- Open Question Answering over Tables and Text cites HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
- Open Question Answering over Tables and Text cites TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Builds on9
- TabFact: A Large-scale Dataset for Table-based Fact VerificationWenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang et al.ICLR 2020 · 674 citations
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular DataPengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian RiedelACL 2020 · 417 citations
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual DataWenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong et al.EMNLP Findings 2020 · 205 citations
- Open Question Answering over Tables and TextWenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang et al.ICLR 2021 · 76 citations
- TGRNet: A Table Graph Reconstruction Network for Table Structure RecognitionWenyuan Xue, Baosheng Yu, Wen Wang, Dacheng Tao et al.ICCV 2021 · 65 citations
Related papers
- SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of SummarizationPhilippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander R. Fabbri et al.EMNLP 2023 · 29 citations
- EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial StatementsIssa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke et al.ICLR 2026 · 9 citations
- TABARD: A Novel Benchmark for Tabular Anomaly Analysis, Reasoning and DetectionManan Roy Choudhury, Anirudh Iyengar Kaniyar Narayana Iyengar, Shikhhar Siingh, Sugeeth Puranam et al.EMNLP Findings 2025
- Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial DocumentsYing He, Zhouhong Gu, Zhecheng Hu, Yubo Zhou et al.ACL Findings 2026
- BizBench: A Quantitative Reasoning Benchmark for Business and FinanceMichael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy et al.ACL 2024 · 10 citations
