ACL Findings2025
Exploring Graph Representations of Logical Forms for Language Modeling
Abstract
We make the case for language models over logical forms (LFLMs), arguing that such models are more data-efficient than their textual counterparts. To that end, we introduce the Graph-based Formal-Logical Distributional Semantics (GFoLDS) prototype, a pretrained LM over graph representations of logical forms, as a proof-of-concept of LFLMs. Using GFoLDS, we present strong experimental evidence that LFLMs can leverage the built-in, basic linguistic knowledge inherent in such models to immediately begin learning more complex patterns. On downstream tasks, we show that GFoLDS vastly outperforms textual, transformer LMs (BERT) pretrained on the same data, indicating that LFLMs can learn with substantially less data than models over plain text. Furthermore, we show that the performance of this model is likely to scale with additional parameters and pretraining data, suggesting the viability of LFLMs in real-world applications.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on9
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Scaling Data-Constrained Language ModelsNiklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao et al.NeurIPS 2023 · 475 citations
- Semantics-Aware BERT for Language UnderstandingZhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li et al.AAAI 2020 · 396 citations
- GraphGen: A Scalable Approach to Domain-agnostic Labeled Graph GenerationNikhil Goyal, Harsh Vardhan Jain, Sayan RanuWWW 2020 · 110 citations
- Scale Efficiently: Insights from Pretraining and Finetuning TransformersYi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus et al.ICLR 2022 · 67 citations
Related papers
- Enhancing Multilingual Language Model with Massive Multilingual Knowledge TriplesLinlin Liu, Xin Li, Ruidan He, Lidong Bing et al.EMNLP 2022 · 15 citations
- MTLS: Making Texts into Linguistic SymbolsWenlong Fei, Xiaohua Wang, Min Hu, Qingyu Zhang et al.EMNLP 2024 · 1 citation
- BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model AlignmentAndrey Sakhovskiy, Elena TutubalinaSIGIR 2025 · 2 citations
- Graph Language ModelsMoritz Plenz, Anette FrankACL 2024
- Language Model Analysis for Ontology Subsumption InferenceYuan He, Jiaoyan Chen, Ernesto Jiménez-Ruiz, Hang Dong et al.ACL Findings 2023 · 12 citations
