Lune

EMNLP Findings2024

Can Large Language Models Understand DL-Lite Ontologies? An Empirical Study

Keyu Wang, Guilin Qi, Jiaqi Li, Songlin Zhai

Abstract

Large language models (LLMs) have shown significant achievements in solving a wide range of tasks. Recently, LLMs' capability to store, retrieve and infer with symbolic knowledge has drawn a great deal of attention, showing their potential to understand structured information. However, it is not yet known whether LLMs can understand Description Logic (DL) ontologies. In this work, we empirically analyze the LLMs' capability of understanding DL-Lite ontologies covering 6 representative tasks from syntactic and semantic aspects. With extensive experiments, we demonstrate both the effectiveness and limitations of LLMs in understanding DL-Lite ontologies. We find that LLMs can understand formal syntax and model-theoretic semantics of concepts and roles quite well. However, LLMs struggle with understanding TBox NI (Negative Inclusion) transitivity and handling ontologies with large ABoxes. We hope that our experiments and analyses provide more insights into LLMs and inspire to build more faithful knowledge engineering solutions.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on6

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni et al.ICLR 2024 · 462 citations
  3. Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical ReasoningLiangming Pan, Alon Albalak, Xinyi Wang, William Yang WangEMNLP Findings 2023 · 145 citations
  4. Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMsSiyuan Wang, Zhongyu Wei, Yejin Choi, Xiang RenACL 2024
  5. Towards Concept-Aware Large Language ModelsChen Shani, Jilles Vreeken, Dafna ShahafEMNLP Findings 2023

Related papers

  1. Do LLMs Really Struggle at NL-FOL Translation? Revealing Their Strengths via a Novel Benchmarking StrategyAndrea Brunello, Luca Geatti, Michele Mignani, Angelo Montanari et al.AAAI 2026
  2. Inductive Learning of Logical Theories with LLMs: A Expressivity-graded AnalysisJoão Pedro Gandarela de Souza, Danilo S. Carvalho, André FreitasAAAI 2025 · 3 citations
  3. Can Large Language Models Understand Symbolic Graphics Programs?Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu et al.ICLR 2025
  4. Out-of-Context Reasoning in Large Language ModelsJonathan Shaki, Emanuele La Malfa, Michael J. Wooldridge, Sarit KrausEMNLP Findings 2025
  5. Do Large Language Models excel in Complex Logical Reasoning with Formal Language?Jin Jiang, Jianing Wang, Yuchen Yan, Yang Liu et al.EMNLP 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines