EMNLP Findings2024
Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study
Keyu Wang, Guilin Qi, Jiaqi Li, Songlin Zhai
Abstract
Large language models (LLMs) have shown significant achievements in solving a wide range of tasks. Recently, LLMs' capability to store, retrieve and infer with symbolic knowledge has drawn a great deal of attention, showing their potential to understand structured information. However, it is not yet known whether LLMs can understand Description Logic (DL) ontologies. In this work, we empirically analyze the LLMs' capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects. With extensive experiments, we demonstrate both the effectiveness and limitations of LLMs in understanding DL-Lite ontologies. We find that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles quite well. However, LLMs struggle with understanding TBox NI (Negative Inclusion) transitivity and handling ontologies with large ABoxes. We hope that our experiments and analyses provide more insights into LLMs and inspire to build more faithful knowledge engineering solutions.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on6
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni et al.ICLR 2024 · 462 citations
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical ReasoningLiangming Pan, Alon Albalak, Xinyi Wang, William Yang WangEMNLP Findings 2023 · 145 citations
- Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMsSiyuan Wang, Zhongyu Wei, Yejin Choi, Xiang RenACL 2024
- Towards Concept-Aware Large Language ModelsChen Shani, Jilles Vreeken, Dafna ShahafEMNLP Findings 2023
Related papers
- Do LLMs Really Struggle at NL-FOL Translation? Revealing Their Strengths via a Novel Benchmarking StrategyAndrea Brunello, Luca Geatti, Michele Mignani, Angelo Montanari et al.AAAI 2026
- Inductive Learning of Logical Theories with LLMs: A Expressivity-graded AnalysisJoão Pedro Gandarela de Souza, Danilo S. Carvalho, André FreitasAAAI 2025 · 3 citations
- Can Large Language Models Understand Symbolic Graphics Programs?Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu et al.ICLR 2025
- Out-of-Context Reasoning in Large Language ModelsJonathan Shaki, Emanuele La Malfa, Michael J. Wooldridge, Sarit KrausEMNLP Findings 2025
- Do Large Language Models excel in Complex Logical Reasoning with Formal Language?Jin Jiang, Jianing Wang, Yuchen Yan, Yang Liu et al.EMNLP 2025 · 3 citations
