LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Study
Dongil Yang, Minjin Kim, Sunghwan Kim, Beong-woo Kwak, Minjun Park, Jinseok Hong, Woontack Woo, Jinyoung Yeo
Abstract
The remarkable reasoning and generalization capabilities of Large Language Models (LLMs) have paved the way for their expanding applications in embodied AI, robotics, and other real-world tasks. To effectively support these applications, grounding in spatial and temporal understanding in multimodal environments is essential. To this end, recent works have leveraged scene graphs, a structured representation that encodes entities, attributes, and their relationships in a scene. However, a comprehensive evaluation of LLMs' ability to utilize scene graphs remains limited. In this work, we introduce Text-Scene Graph (TSG) Bench, a benchmark designed to systematically assess LLMs' ability to (1) understand scene graphs and (2) generate them from textual narratives. With TSG Bench we evaluate 11 LLMs and reveal that, while models perform well on scene graph understanding, they struggle with scene graph generation, particularly for complex narratives. Our analysis indicates that these models fail to effectively decompose discrete scenes from a complex narrative, leading to a bottleneck when generating scene graphs. These findings underscore the need for improved methodologies in scene graph generation and provide valuable insights for future research. The demonstration of our benchmark is available at https://tsg-bench.netlify.app. Additionally, our code and evaluation data are publicly available at https://github.com/docworlds/tsg-bench.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement cites FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing
- Ego4D: Around the World in 3, 000 Hours of Egocentric Video cites Language Models are Few-Shot Learners
Cited by top-tier papers6
- DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph RefinementShaoqing Lin, Chong Teng, Fei Li, Donghong Ji et al.EMNLP 2025
- Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video UnderstandingKe Ma, Jiaqi Tang, Bin Guo, Xueting Han et al.ACL 2026
- BUSSARD: Normalizing Flows for Bijective Universal Scene-Specific Anomalous Relationship DetectionMelissa Schween, Mathis Kruse, Bodo RosenhahnCVPR 2026
- GRASP: Graph Reasoning via Agentic Solving and Probing of LLMsXiaojun Guo, Mingxue Tian, Chenheng Zhang, Xiaohan Wang et al.ICML 2026
- KG-ViP: Bridging Knowledge Grounding and Visual Perception in Multi-modal LLMs for Visual Question AnsweringZhiyang Li, Ao Ke, Yukun Cao, Xike XieACL 2026
Builds on9
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Ego4D: Around the World in 3, 000 Hours of Egocentric VideoKristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis et al.CVPR 2022 · 525 citations
- 3D Scene Graph: A Structure for Unified Semantics, 3D Space, and CameraIro Armeni, Zhi-Yang He, Amir Zamir, JunYoung Gwak et al.ICCV 2019 · 474 citations
- SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object NavigationHang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou et al.NeurIPS 2024 · 215 citations
- FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph ParsingZhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu et al.ACL Findings 2023 · 19 citations
Related papers
- ESCA: Contextualizing Embodied Agents via Scene-Graph GenerationJiani Huang, Amish Sethi, Matthew Kuo, Mayank Keoliya et al.NeurIPS 2025 · 7 citations
- STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?Yun Li, Yiming Zhang, Tao Lin, XiangRui Liu et al.ICCV 2025 · 4 citations
- 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene UnderstandingTatiana Zemskova, Dmitry A. YudinICCV 2025 · 7 citations
- TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image UnderstandingKun Zhang, Liqiang Niu, Zhen Cao, Fandong Meng et al.EMNLP Findings 2025
- HIS-GPT: Towards 3D Human-In-Scene Multimodal UnderstandingJiahe Zhao, Ruibing Hou, Zejie Tian, Hong Chang et al.ICCV 2025 · 6 citations
