Lune

EMNLP Findings2021

Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers

Shane Storks, Joyce Chai

Abstract

As large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, statistical bias in benchmark data and probing studies have recently called into question their true capabilities.For a more informative evaluation than accuracy on text classification tasks can offer, we propose evaluating systems through a novel measure of prediction coherence.We apply our framework to two existing language understanding benchmarks with different properties to demonstrate its versatility.Our experimental results show that this evaluation framework, although simple in ideas and implementation, is a quick, effective, and versatile measure to provide insight into the coherence of machines' predictions. Story 1:Kelly wanted to try out for soccer this year.Kelly tried

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Deberta: decoding-Enhanced Bert with Disentangled Attention cites Language Models are Few-Shot Learners
  • From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning cites Language Models are Few-Shot Learners
  • CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks cites Abductive Commonsense Reasoning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. CRoW: Benchmarking Commonsense Reasoning in Real-World TasksMete Ismayilzada, Debjit Paul, Syrielle Montariol, Mor Geva et al.EMNLP 2023 · 3 citations
  2. From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense ReasoningZheyuan Zhang, Shane Storks, Fengyuan Hu, Sungryull Sohn et al.EMNLP 2023

Builds on7

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  3. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  4. Abductive Commonsense ReasoningChandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi et al.ICLR 2020 · 521 citations
  5. Beyond Accuracy: Behavioral Testing of NLP Models with CheckListMarco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, Sameer SinghACL 2020 · 51 citations

Related papers

  1. Coherency Improved Explainable Recommendation via Large Language ModelShijie Liu, Ruixin Ding, Weihai Lu, Jun Wang et al.AAAI 2025 · 10 citations
  2. Incoherence as Oracle-less Measure of Error in LLM-Based Code GenerationThomas Jean-Michel Valentin, Ardi Madadi, Gaetano Sapia, Marcel BöhmeAAAI 2026 · 4 citations
  3. Co-Eval: Augmenting LLM-based Evaluation with Machine MetricsLing-I Wu, Weijie Wu, Minyu Chen, Jianxin Xue et al.EMNLP 2025
  4. Large Language Models for Predictive Analysis: How Far Are They?Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang ShiACL Findings 2025 · 2 citations
  5. Towards Quantifiable Dialogue Coherence EvaluationZheng Ye, Liucun Lu, Lishan Huang, Liang Lin et al.ACL 2021
Dusk over the sea between two cliffs drawn in fine vertical lines