Lune

EMNLP Findings2024

CLEAR: Can Language Models Really Understand Causal Graphs?

Sirui Chen, Mengying Xu, Kun Wang, Xingyu Zeng, Rui Zhao, Shengjie Zhao, Chaochao Lu

Abstract

Causal reasoning is a cornerstone of how humans interpret the world. To model and reason about causality, causal graphs offer a concise yet effective solution. Given the impressive advancements in language models, a crucial question arises: can they really understand causal graphs? To this end, we pioneer an investigation into language models' understanding of causal graphs. Specifically, we develop a framework to define causal graph understanding, by assessing language models' behaviors through four practical criteria derived from diverse disciplines (e.g., philosophy and psychology). We then develop CLEAR, a novel benchmark that defines three complexity levels and encompasses 20 causal graph-based tasks across these levels. Finally, based on our framework and benchmark, we conduct extensive experiments on six leading language models and summarize five empirical findings.

Our results indicate that while language models demonstrate a preliminary understanding of causal graphs, significant potential for improvement remains. Our project website is at https: //github.com/OpenCausaLab/CLEAR.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites Language Models are Few-Shot Learners
  • Graph of Thoughts: Solving Elaborate Problems with Large Language Models cites Language Models are Few-Shot Learners
  • Self-Instruct: Aligning Language Models with Self-Generated Instructions cites Language Models are Few-Shot Learners
  • Talk like a Graph: Encoding Graphs for Large Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Causal Discovery through Synergizing Large Language Model and Data-Driven ReasoningHuaming Du, Yujia Zheng, Baoyu Jing, Yu Zhao et al.KDD 2025 · 1 citation
  2. Teaching Transformers Causal Reasoning through Axiomatic TrainingAniket Vashishtha, Abhinav Kumar, Atharva Pandey, Abbavaram Gowtham Reddy et al.ICML 2025
  3. Counterfactual Planning for Generalizable Agents' ActionsJiarun Fu, Lizhong Ding, Qiuning Wei, Yuhan Guo et al.AAAI 2026

Builds on21

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
  3. Graph of Thoughts: Solving Elaborate Problems with Large Language ModelsMaciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger et al.AAAI 2024 · 1,292 citations
  4. Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu et al.ACL 2023 · 540 citations
  5. Talk like a Graph: Encoding Graphs for Large Language ModelsBahare Fatemi, Jonathan Halcrow, Bryan PerozziICLR 2024 · 194 citations

Related papers

  1. CELLO: Causal Evaluation of Large Vision-Language ModelsMeiqi Chen, Bo Peng, Yan Zhang, Chaochao LuEMNLP 2024 · 4 citations
  2. CausalLink: An Interactive Evaluation Framework for Causal ReasoningJinyue Feng, Frank RudziczACL Findings 2025
  3. CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language ModelsAneesh Komanduri, Karuna Bhaila, Xintao WuEMNLP 2025
  4. CausalityCheck: A Framework for Evaluating Causal Reasoning in Large Language ModelsJiang Li, Zehua Duo, Guanglai Gao, Xiangdong SuACL Findings 2026
  5. COLD: Causal reasOning in cLosed Daily activitiesAbhinav Joshi, Areeb Ahmad, Ashutosh ModiNeurIPS 2024 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines