Lune

EMNLP Findings2024

A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles

Eun-Kyoung Rosa Lee, Sathvik Nair, Naomi Feldman

Abstract

We present a systematic evaluation of large language models' sensitivity to argument roles, i.e., who did what to whom, by replicating psycholinguistic studies on human argument role processing. In three experiments, we find that language models are able to distinguish verbs that appear in plausible and implausible contexts, where plausibility is determined through the relation between the verb and its preceding arguments. However, none of the models capture the same selective patterns that human comprehenders exhibit during real-time verb prediction. This indicates that language models' capacity to detect verb plausibility does not arise from the same mechanism that underlies human real-time sentence processing.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMsKarin de Langis, Jong Inn Park, Andreas Schramm, Bin Hu et al.ACL 2025

Builds on10

  1. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  2. Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
  3. Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training TokensByung-Doh Oh, William SchulerEMNLP Findings 2023 · 21 citations
  4. The Linearity of the Effect of Surprisal on Reading Times across LanguagesWeijie Xu, Jason Chon, Tianran Liu, Richard FutrellEMNLP Findings 2023 · 11 citations
  5. A Language Model with Limited Memory Capacity Captures Interference in Human Sentence ProcessingWilliam Timkey, Tal LinzenEMNLP Findings 2023 · 6 citations

Related papers

  1. Plausibility Processing in Transformer Language Models: Focusing on the Role of Attention Heads in GPTSoo RyuEMNLP Findings 2023
  2. A fine-grained comparison of pragmatic language understanding in humans and language modelsJennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko et al.ACL 2023 · 45 citations
  3. John praised Mary because he? Implicit Causality Bias and Its Interaction with Explicit Cues in LMsYova Kementchedjhieva, Mark Anderson, Anders SøgaardACL Findings 2021 · 1 citation
  4. Decision Biases and Intent-Irony Decoupling in Large Language ModelsKewei Guo, Lingyun Sun, Manhao GuanACL Findings 2026
  5. The Thin Line Between Comprehension and Persuasion in LLMsAdrian de Wynter, Tangming YuanACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines