ACL Findings2024
Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction
Jinwook Park, Kangil Kim
Abstract
Neural parameterization has significantly advanced unsupervised grammar induction. However, training these models with a traditional likelihood loss for all possible parses exacerbates two issues: 1) structural optimization ambiguity that arbitrarily selects one among structurally ambiguous optimal grammars despite the specific preference of gold parses, and 2) structural simplicity bias that leads a model to underutilize rules to compose parse trees. These challenges subject unsupervised neural grammar induction (UNGI) to inevitable prediction errors, high variance, and the necessity for extensive grammars to achieve accurate predictions. This paper tackles these issues, offering a comprehensive analysis of their origins. As a solution, we introduce sentencewise parse-focusing to reduce the parse pool per sentence for loss evaluation, using the structural bias from pre-trained parsers on the same dataset. In unsupervised parsing benchmark tests, our method significantly improves performance while effectively reducing variance and bias toward overly simplistic parses. Our research promotes learning more compact, accurate, and consistent explicit grammars, facilitating better interpretability.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Ensemble Distillation for Unsupervised Constituency Parsing cites Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders
- Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Induction cites Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders
- Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Induction cites Neural Bi-Lexicalized PCFG Induction
- Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Induction cites StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
- Neural Bi-Lexicalized PCFG Induction cites Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders
- Neural Bi-Lexicalized PCFG Induction cites StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
Cited by top-tier papers1
Builds on4
- Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive AutoencodersAndrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman et al.EMNLP 2020 · 27 citations
- Ensemble Distillation for Unsupervised Constituency ParsingBehzad Shayegh, Yanshuai Cao, Xiaodan Zhu, Jackie C. K. Cheung et al.ICLR 2024 · 10 citations
- Neural Bi-Lexicalized PCFG InductionSonglin Yang, Yanpeng Zhao, Kewei TuACL 2021
- StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language ModelingYikang Shen, Yi Tay, Che Zheng, Dara Bahri et al.ACL 2021
Related papers
- Leveraging Grammar Induction for Language Understanding and GenerationJushi Kai, Shengyuan Hou, Yusheng Huang, Zhouhan LinEMNLP Findings 2024 · 1 citation
- On Eliciting Syntax from Language Models via HashingYiran Wang, Masao UtiyamaEMNLP 2024
- Unsupervised Parsing via Constituency TestsSteven Cao, Nikita Kitaev, Dan KleinEMNLP 2020 · 25 citations
- Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at ScaleXiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu et al.ACL 2024 · 1 citation
- Improving Unsupervised Constituency Parsing via Maximizing Semantic InformationJunjie Chen, Xiangheng He, Yusuke Miyao, Danushka BollegalaICLR 2025
