Lune

EMNLP Findings2021

Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models

Taeuk Kim, Bowen Li, Sang-goo Lee

Abstract

As it has been unveiled that pre-trained language models (PLMs) are to some extent capable of recognizing syntactic concepts in natural language, much effort has been made to develop a method for extracting complete (binary) parses from PLMs without training separate parsers. We improve upon this paradigm by proposing a novel chart-based method and an effective top-K ensemble technique. Moreover, we demonstrate that we can broaden the scope of application of the approach into multilingual settings. Specifically, we show that by applying our method on multilingual PLMs, it becomes possible to induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, attaining performance superior or comparable to that of unsupervised PCFGs. We also verify that our approach is robust to cross-lingual transfer. Finally, we provide analyses on the inner workings of our method. For instance, we discover universal attention heads which are consistently sensitive to syntactic information irrespective of the input language.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers cites Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
  • Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers cites Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
  • Finding Universal Grammatical Relations in Multilingual BERT cites Unsupervised Cross-lingual Representation Learning at Scale
  • Finding Universal Grammatical Relations in Multilingual BERT cites Multilingual Alignment of Contextual Word Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Contextual Distortion Reveals Constituency: Masked Language Models are Implicit ParsersJiaxi Li, Wei LuACL 2023 · 3 citations

Builds on5

  1. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  2. Multilingual Alignment of Contextual Word RepresentationsSteven Cao, Nikita Kitaev, Dan KleinICLR 2020 · 211 citations
  3. Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTZhiyong Wu, Yun Chen, Ben Kao, Qun LiuACL 2020 · 158 citations
  4. Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar InductionTaeuk Kim, Jihun Choi, Daniel Edmiston, Sang-goo LeeICLR 2020 · 92 citations
  5. Finding Universal Grammatical Relations in Multilingual BERTEthan A. Chi, John Hewitt, Christopher D. ManningACL 2020 · 7 citations

Related papers

  1. Multilingual Pre-training with Universal Dependency LearningKailai Sun, Zuchao Li, Hai ZhaoNeurIPS 2021 · 11 citations
  2. Investigating Non-local Features for Neural Constituency ParsingLeyang Cui, Sen Yang, Yue ZhangACL 2022
  3. Constrained Language Models Yield Few-Shot Semantic ParsersRichard Shin, Christopher H. Lin, Sam Thomson, Charles Chen et al.EMNLP 2021 · 131 citations
  4. Unveiling the Black Box of PLMs with Semantic Anchors: Towards Interpretable Neural Semantic ParsingLunyiu Nie, Jiuding Sun, Yanlin Wang, Lun Du et al.AAAI 2023 · 9 citations
  5. On Eliciting Syntax from Language Models via HashingYiran Wang, Masao UtiyamaEMNLP 2024
Dusk over the sea between two cliffs drawn in fine vertical lines