Lune

ACL Findings2025

On the Role of Semantic Proto-roles in Semantic Analysis: What do LLMs know about agency?

Elizabeth Spaulding, Shafiuddin Rehan Ahmed, James H. Martin

Abstract

Large language models (LLMs) are increasingly used in decision-making contexts, yet their ability to reason over event structure-an important component in the situational awareness needed to make complex decisions-is not well understood. By operationalizing protorole theory, which characterizes agents via properties such as instigation and volition and patients via properties such as change of state, we examine the ability of LLMs to answer questions that require complex, multi-step event reasoning. Specifically, we investigate the extent to which LLMs capture semantic roles such as "agent" and "patient" through zero-shot prompts, and whether incorporating semantic proto-role labeling (SPRL) context improves semantic role labeling (SRL) performance in a zero-shot setting. We find that, while SPRL context sometimes degrades SRL accuracy in high-performing models (e.g., GPT-4o), it also uncovers an internal consistency between SPRL and SRL predictions that mirrors linguistic theory, and provides evidence that LLMs implicitly encode consistent multi-dimensional event role knowledge. Furthermore, our experiments support prior work showing that LLMs underperform human annotators in complex semantic analysis.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OLMo: Accelerating the Science of Language Models cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
  • What's In My Big Data? cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
  2. What's In My Big Data?Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander et al.ICLR 2024 · 135 citations
  3. Better Combine Them Together! Integrating Syntactic Constituency and Dependency Representations for Semantic Role LabelingHao Fei, Shengqiong Wu, Yafeng Ren, Fei Li et al.ACL Findings 2021 · 82 citations
  4. OLMo: Accelerating the Science of Language ModelsDirk Groeneveld, Iz Beltagy, Evan Pete Walsh, Akshita Bhagia et al.ACL 2024 · 52 citations
  5. "You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning RepresentationAllyson Ettinger, Jena D. Hwang, Valentina Pyatkin, Chandra Bhagavatula et al.EMNLP Findings 2023 · 8 citations

Related papers

  1. Advancing Reasoning with Off-the-Shelf LLMs: A Semantic Structure PerspectivePengfei He, Zitao Li, Yue Xing, Yaliang Li et al.EMNLP Findings 2025
  2. LLMs Can Also Do Well! Breaking Barriers in Semantic Role Labeling via Large Language ModelsXinxin Li, Huiyao Chen, Chengjun Liu, Jing Li et al.ACL Findings 2025 · 3 citations
  3. Probing for Predicate Argument Structures in Pretrained Language ModelsSimone Conia, Roberto NavigliACL 2022
  4. The Curious Case of ControlElias Stengel-Eskin, Benjamin Van DurmeEMNLP 2022
  5. Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?Haoang Chi, He Li, Wenjing Yang, Feng Liu et al.NeurIPS 2024 · 124 citations
Dusk over the sea between two cliffs drawn in fine vertical lines