Lune

EMNLP Findings2022

Autoregressive Structured Prediction with Language Models

Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, Mrinmaya Sachan

Abstract

In recent years, NLP has moved towards the application of language models to a more diverse set of tasks. However, applying language models to structured prediction, e.g., predicting parse trees, taggings, and coreference chains, is not straightforward. Prior work on language model-based structured prediction typically flattens the target structure into a string to easily fit it into the language modeling framework. Such flattening limits the accessibility of structural information and can lead to inferior performance compared to approaches that overtly model the structure. In this work, we propose to construct a conditional language model over sequences of structure-building actions, rather than over strings in a way that makes it easier for the model to pick up on intra-structure dependencies. Our method sets the new state of the art on named entity recognition, end-to-end relation extraction, and coreference resolution.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Autoregressive Entity Retrieval cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks cites Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders
  • Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks cites Packed Levitated Marker for Entity and Relation Extraction
  • Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends cites CorefQA: Coreference Resolution as Query-based Span Prediction
  • Packed Levitated Marker for Entity and Relation Extraction cites Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders
  • Packed Levitated Marker for Entity and Relation Extraction cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Packed Levitated Marker for Entity and Relation Extraction cites CorefQA: Coreference Resolution as Query-based Span Prediction
  • An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction cites Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders
  • An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction cites Autoregressive Entity Retrieval
  • An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. An Autoregressive Text-to-Graph Framework for Joint Entity and Relation ExtractionUrchade Zaratiana, Nadi Tomeh, Pierre Holat, Thierry CharnoisAAAI 2024 · 39 citations
  2. Structured Prediction with Stronger Consistency GuaranteesAnqi Mao, Mehryar Mohri, Yutao ZhongNeurIPS 2023 · 37 citations
  3. Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural NetworksZhaohui Yan, Songlin Yang, Wei Liu, Kewei TuEMNLP 2023 · 20 citations
  4. PAC Prediction Sets for Large Language Models of CodeAdam Khakhar, Stephen Mell, Osbert BastaniICML 2023 · 9 citations
  5. Maverick: Efficient and Accurate Coreference Resolution Defying Recent TrendsGiuliano Martinelli, Edoardo Barba, Roberto NavigliACL 2024 · 8 citations

Builds on6

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence EncodersJue Wang, Wei LuEMNLP 2020 · 209 citations
  3. Autoregressive Entity RetrievalNicola De Cao, Gautier Izacard, Sebastian Riedel, Fabio PetroniICLR 2021 · 200 citations
  4. CorefQA: Coreference Resolution as Query-based Span PredictionWei Wu, Fei Wang, Arianna Yuan, Fei Wu et al.ACL 2020 · 153 citations
  5. Packed Levitated Marker for Entity and Relation ExtractionDeming Ye, Yankai Lin, Peng Li, Maosong SunACL 2022 · 140 citations

Related papers

  1. Structured Prediction as Translation between Augmented Natural LanguagesGiovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma et al.ICLR 2021 · 351 citations
  2. DeepStruct: Pretraining of Language Models for Structure PredictionChenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong et al.ACL Findings 2022 · 78 citations
  3. Nested Named Entity Recognition as Single-Pass Sequence LabelingAlberto Muñoz-Ortiz, David Vilares, Caio COrro, Carlos Gómez-RodríguezEMNLP Findings 2025 · 1 citation
  4. Prompting Language Models for Linguistic StructureTerra Blevins, Hila Gonen, Luke ZettlemoyerACL 2023 · 15 citations
  5. Attending to Entities for Better Text UnderstandingPengxiang Cheng, Katrin ErkAAAI 2020 · 42 citations
Dusk over the sea between two cliffs drawn in fine vertical lines