On Eliciting Syntax from Language Models via Hashing
Abstract
Unsupervised parsing, also known as grammar induction, aims to infer syntactic structure from raw text. Recently, binary representation has exhibited remarkable informationpreserving capabilities at both lexicon and syntax levels. In this paper, we explore the possibility of leveraging this capability to deduce parsing trees from raw text, relying solely on the implicitly induced grammars within models. To achieve this, we upgrade the bit-level CKY from zero-order to first-order to encode the lexicon and syntax in a unified binary representation space, switch training from supervised to unsupervised under the contrastive hashing framework, and introduce a novel loss function to impose stronger yet balanced alignment signals. Our model 1 shows competitive performance on various datasets, therefore, we claim that our method is effective and efficient enough to acquire high-quality parsing trees from pre-trained language models at a low cost.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on15
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Revisiting Pre-Trained Models for Chinese Natural Language ProcessingYiming Cui, Wanxiang Che, Ting Liu, Bing Qin et al.EMNLP Findings 2020 · 866 citations
- Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar InductionTaeuk Kim, Jihun Choi, Daniel Edmiston, Sang-goo LeeICLR 2020 · 92 citations
- Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive AutoencodersAndrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman et al.EMNLP 2020 · 27 citations
Related papers
- To be Continuous, or to be Discrete, Those are Bits of QuestionsYiran Wang, Masao UtiyamaACL 2024 · 3 citations
- Leveraging Grammar Induction for Language Understanding and GenerationJushi Kai, Shengyuan Hou, Yusheng Huang, Zhouhan LinEMNLP Findings 2024 · 1 citation
- StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language ModelingYikang Shen, Yi Tay, Che Zheng, Dara Bahri et al.ACL 2021
- Improving Unsupervised Constituency Parsing via Maximizing Semantic InformationJunjie Chen, Xiangheng He, Yusuke Miyao, Danushka BollegalaICLR 2025
- Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language ModelsTaeuk Kim, Bowen Li, Sang-goo LeeEMNLP Findings 2021 · 5 citations
