EMNLP Findings2021
Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models
Abstract
As it has been unveiled that pre-trained language models (PLMs) are to some extent capable of recognizing syntactic concepts in natural language, much effort has been made to develop a method for extracting complete (binary) parses from PLMs without training separate parsers. We improve upon this paradigm by proposing a novel chart-based method and an effective top-K ensemble technique. Moreover, we demonstrate that we can broaden the scope of application of the approach into multilingual settings. Specifically, we show that by applying our method on multilingual PLMs, it becomes possible to induce non-trivial parses for sentences from nine languages in an integrated and language-agnostic manner, attaining performance superior or comparable to that of unsupervised PCFGs. We also verify that our approach is robust to cross-lingual transfer. Finally, we provide analyses on the inner workings of our method. For instance, we discover universal attention heads which are consistently sensitive to syntactic information irrespective of the input language.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers cites Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT
- Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers cites Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
- Finding Universal Grammatical Relations in Multilingual BERT cites Unsupervised Cross-lingual Representation Learning at Scale
- Finding Universal Grammatical Relations in Multilingual BERT cites Multilingual Alignment of Contextual Word Representations
Cited by top-tier papers1
Builds on5
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
- Multilingual Alignment of Contextual Word RepresentationsSteven Cao, Nikita Kitaev, Dan KleinICLR 2020 · 211 citations
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTZhiyong Wu, Yun Chen, Ben Kao, Qun LiuACL 2020 · 158 citations
- Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar InductionTaeuk Kim, Jihun Choi, Daniel Edmiston, Sang-goo LeeICLR 2020 · 92 citations
- Finding Universal Grammatical Relations in Multilingual BERTEthan A. Chi, John Hewitt, Christopher D. ManningACL 2020 · 7 citations
Related papers
- Multilingual Pre-training with Universal Dependency LearningKailai Sun, Zuchao Li, Hai ZhaoNeurIPS 2021 · 11 citations
- Investigating Non-local Features for Neural Constituency ParsingLeyang Cui, Sen Yang, Yue ZhangACL 2022
- Constrained Language Models Yield Few-Shot Semantic ParsersRichard Shin, Christopher H. Lin, Sam Thomson, Charles Chen et al.EMNLP 2021 · 131 citations
- Unveiling the Black Box of PLMs with Semantic Anchors: Towards Interpretable Neural Semantic ParsingLunyiu Nie, Jiuding Sun, Yanlin Wang, Lun Du et al.AAAI 2023 · 9 citations
- On Eliciting Syntax from Language Models via HashingYiran Wang, Masao UtiyamaEMNLP 2024
