Lune

EMNLP Findings2021

Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding

Shane Storks, Qiaozi Gao, Yichi Zhang, Joyce Chai

Abstract

Large-scale, pre-trained language models (LMs) have achieved human-level performance on a breadth of language understanding tasks. However, evaluations only based on end task performance shed little light on machines' true ability in language understanding and reasoning. In this paper, we highlight the importance of evaluating the underlying reasoning process in addition to end performance. Toward this goal, we introduce Tiered Reasoning for Intuitive Physics (TRIP), a novel commonsense reasoning dataset with dense annotations that enable multi-tiered evaluation of machines' reasoning process. Our empirical results show that while large LMs can achieve high end performance, they struggle to support their predictions with valid supporting evidence. The TRIP dataset and our baseline results will motivate verifiable evaluation of commonsense reasoning and facilitate future research toward developing better language understanding and reasoning models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Deberta: decoding-Enhanced Bert with Disentangled Attention cites Language Models are Few-Shot Learners
  • ReSeeding Latent States for Sequential Language Understanding cites Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
  • Language Models are Few-Shot Learners cites Adversarial NLI: A New Benchmark for Natural Language Understanding
  • McOmet: Multimodal Fusion Transformer for Physical Audiovisual Commonsense Reasoning cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • McOmet: Multimodal Fusion Transformer for Physical Audiovisual Commonsense Reasoning cites Deberta: decoding-Enhanced Bert with Disentangled Attention
  • CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • The Generative AI Paradox: "What It Can Create, It May Not Understand" cites PIQA: Reasoning about Physical Commonsense in Natural Language
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. The Generative AI Paradox: "What It Can Create, It May Not Understand"Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman et al.ICLR 2024 · 116 citations
  2. McOmet: Multimodal Fusion Transformer for Physical Audiovisual Commonsense ReasoningDaoming Zong, Shiliang SunAAAI 2023 · 5 citations
  3. Breakpoint Transformers for Modeling and Tracking Intermediate BeliefsKyle Richardson, Ronen Tamari, Oren Sultan, Dafna Shahaf et al.EMNLP 2022 · 5 citations
  4. CRoW: Benchmarking Commonsense Reasoning in Real-World TasksMete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva et al.EMNLP 2023 · 3 citations
  5. ReSeeding Latent States for Sequential Language UnderstandingStéphane Aroca-Ouellette, Katharina von der Wense, Alessandro RonconeEMNLP 2025 · 1 citation

Builds on9

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  3. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  4. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataEmily M. Bender, Alexander KollerACL 2020 · 914 citations
  5. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations

Related papers

  1. From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense ReasoningZheyuan Zhang, Shane Storks, Fengyuan Hu, Sungryull Sohn et al.EMNLP 2023
  2. ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question AnsweringFrancesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia et al.ACL 2026 · 1 citation
  3. ExpliCa: Evaluating Explicit Causal Reasoning in Large Language ModelsMartina Miliani, Serena Auriemma, Alessandro Bondielli, Emmanuele Chersoni et al.ACL Findings 2025 · 4 citations
  4. WikiWhy: Answering and Explaining Cause-and-Effect QuestionsMatthew Ho, Aditya Sharma, Justin Chang, Michael Saxon et al.ICLR 2023 · 8 citations
  5. SpaCE-Eval: A Benchmark for Real-World Multi-Modal ReasoningXuyou Yang, Yucheng Zhao, Wenxuan Zhang, Immanuel KohICLR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines