Lune

EMNLP Findings2023

"You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation

Allyson Ettinger, Jena D. Hwang, Valentina Pyatkin, Chandra Bhagavatula, Yejin Choi

Abstract

Large language models (LLMs) show amazing proficiency and fluency in the use of language. Does this mean that they have also acquired insightful linguistic knowledge about the language, to an extent that they can serve as an "expert linguistic annotator"? In this paper, we examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning structure, focusing on the Abstract Meaning Representation (AMR; Banarescu et al. 2013) parsing formalism, which provides rich graphical representations of sentence meaning structure while abstracting away from surface forms. We compare models' analysis of this semantic structure across two settings: 1) direct production of AMR parses based on zero-and few-shot prompts, and 2) indirect partial reconstruction of AMR via metalinguistic natural language queries (e.g., "Identify the primary event of this sentence, and the predicate corresponding to that event."). Across these settings, we find that models can reliably reproduce the basic format of AMR, and can often capture core event, argument, and modifier structure-however, model outputs are prone to frequent and major errors, and holistic analysis of parse acceptability shows that even with few-shot demonstrations, models have virtually 0% success in producing fully accurate parses. Eliciting natural language responses produces similar patterns of errors. Overall, our findings indicate that these models out-of-the-box can capture aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions cites One SPRING to Rule Them Both: Symmetric AMR Semantic Parsing and Generation without a Complex Pipeline
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Multi-Modal Semantic Parsing for the Interpretation of Tombstone InscriptionsXiao Zhang, Johan BosACM MM 2025

Builds on3

  1. One SPRING to Rule Them Both: Symmetric AMR Semantic Parsing and Generation without a Complex PipelineMichele Bevilacqua, Rexhina Blloshmi, Roberto NavigliAAAI 2021 · 197 citations
  2. Constrained Language Models Yield Few-Shot Semantic ParsersRichard Shin, Christopher H. Lin, Sam Thomson, Charles Chen et al.EMNLP 2021 · 131 citations
  3. Graph Pre-training for AMR Parsing and GenerationXuefeng Bai, Yulong Chen, Yue ZhangACL 2022

Related papers

  1. SR-LLM: Rethinking the Structured Representation in Large Language ModelJiahuan Zhang, Tianheng Wang, Ziyi Huang, Yulong Wu et al.ACL 2025 · 5 citations
  2. Semantic Representation for Dialogue ModelingXuefeng Bai, Yulong Chen, Linfeng Song, Yue ZhangACL 2021
  3. Incorporating Graph Information in Transformer-based AMR ParsingPavlo Vasylenko, Pere-Lluís Huguet Cabot, Abelardo Carlos Martinez Lorenzo, Roberto NavigliACL Findings 2023 · 7 citations
  4. Improving Long Dialogue Summarization with Semantic Graph RepresentationBobby Yilun Hua, Zhaoyuan Deng, Kathleen R. McKeownACL Findings 2023 · 7 citations
  5. XL-AMR: Enabling Cross-Lingual AMR Parsing with Transfer Learning TechniquesRexhina Blloshmi, Rocco Tripodi, Roberto NavigliEMNLP 2020 · 48 citations
Dusk over the sea between two cliffs drawn in fine vertical lines