EMNLP Findings2025
FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking
Abstract
The field of explainable Automatic Fact-Checking (AFC) aims to enhance the transparency and trustworthiness of automated factverification systems by providing clear and comprehensible explanations.However, the effectiveness of these explanations depends on their actionability-the extent to which an AFC explanation pinpoints the error, supplies the correct fact, and backs it with sources.Despite actionability being critical for high-quality explanations, no prior research has proposed a method to evaluate it.This paper introduces FinGrAct, a fine-grained evaluation framework that can access the web and is designed to assess actionability in AFC explanations through well-defined criteria.We also introduce a novel dataset to evaluate actionability in AFC explanations.FinGrAct surpasses state-of-theart (SOTA) evaluators, achieving the highest Pearson and Kendall correlation with human judgments while demonstrating the lowest egocentric bias, making it a more robust evaluation approach for actionability evaluation in AFC. Label: False Explanation 1 (full actionability):The claim has two errors in earth's description.The errors are in the words 'flat' and 'red'.The correct version of the claim is : "Earth is round and blue".Check NASA at:
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization cites G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models cites BLEURT: Learning Robust Metrics for Text Generation
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models cites FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
Builds on12
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language ModelsSeungone Kim, Jamin Shin, Yejin Choi, Joel Jang et al.ICLR 2024 · 468 citations
- FLASK: Fine-grained Language Model Evaluation based on Alignment Skill SetsSeonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang et al.ICLR 2024 · 176 citations
- Large Language Models are Not Yet Human-Level Evaluators for Abstractive SummarizationChenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You et al.EMNLP Findings 2023 · 52 citations
- BLEURT: Learning Robust Metrics for Text GenerationThibault Sellam, Dipanjan Das, Ankur P. ParikhACL 2020 · 40 citations
Related papers
- Automated Justification Production for Claim Veracity in Fact Checking: A Survey on Architectures and ApproachesIslam Eldifrawi, Shengrui Wang, Amine TrabelsiACL 2024 · 4 citations
- Generating Media Background Checks for Automated Source Critical ReasoningMichael Sejr SchlichtkrullEMNLP Findings 2024 · 2 citations
- FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question AnsweringAnku Rani, S. M. Towhidul Islam Tonmoy, Dwip Dalal, Shreya Gautam et al.ACL 2023 · 16 citations
- Interpretable Automatic Fine-grained Inconsistency Detection in Text SummarizationHou Pong Chan, Qi Zeng, Heng JiACL Findings 2023 · 8 citations
- Explainable Automated Fact-Checking for Public Health ClaimsNeema Kotonya, Francesca ToniEMNLP 2020 · 10 citations
