Lune

ICLR2025Top-tier venue

PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding

Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Campagnolo Guizilini, Yue Wang

Abstract

Understanding the physical world is a fundamental challenge in embodied AI, critical for enabling agents to perform complex tasks and operate safely in real-world environments. While Vision-Language Models (VLMs) have shown great promise in reasoning and task planning for embodied agents, their ability to comprehend physical phenomena remains extremely limited. To close this gap, we introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs' physical world understanding capability across a diverse set of tasks. PhysBench contains 10,002 entries of interleaved video-image-text data, categorized into four major domains: physical object properties, physical object relationships, physical scene understanding, and physics-based dynamics, further divided into 19 subclasses and 8 distinct capability dimensions. Our extensive experiments, conducted on 75 representative VLMs, reveal that while these models excel in common-sense reasoning, they struggle with understanding the physical world -- likely due to the absence of physical knowledge in their training data and the lack of embedded physical priors. To tackle the shortfall, we introduce PhysAgent, a novel framework that combines the generalization strengths of VLMs with the specialized expertise of vision models, significantly enhancing VLMs' physical understanding across a variety of tasks, including an 18.4% improvement on GPT-4o. Furthermore, our results demonstrate that enhancing VLMs' physical world understanding capabilities can help embodied agents such as MOKA. We believe that PhysBench and PhysAgent offer valuable insights and contribute to bridging the gap between VLMs and physical world understanding.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

36 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LEXam: Benchmarking Legal Reasoning on 340 Law Exams cites PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers36

  1. OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language ModelsMengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang et al.ICLR 2026 · 109 citations
  2. More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning ModelsZhongxing Xu, Chengzhi Liu, Qingyue Wei, Juncheng Wu et al.NeurIPS 2025 · 103 citations
  3. LEXam: Benchmarking Legal Reasoning on 340 Law ExamsYu Fan, Jingwei Ni, Jakob Merane, Yang Tian et al.ICLR 2026 · 56 citations
  4. PhysReason: A Comprehensive Benchmark towards Physics-Based ReasoningXinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang et al.ACL 2025 · 51 citations
  5. Evaluating Newtonian Mechanics in Video Generative Models with Real Physical SystemsAntonios Tragoudaras, Chenyu Zhang, Daniil Cherniavskii, Antonis Vozikis et al.ICML 2026 · 39 citations

Related papers

  1. PAI-Bench: A Comprehensive Benchmark For Physical AIFengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan et al.CVPR 2026 · 32 citations
  2. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied AgentsRui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao et al.ICML 2025
  3. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video GenerationFanqing Meng, Jiaqi Liao, Xinyu Tan, Quanfeng Lu et al.ICML 2025
  4. PhysVLM: Enabling Visual Language Models to Understand Robotic Physical ReachabilityWeijie Zhou, Manli Tao, Chaoyang Zhao, Haiyun Guo et al.CVPR 2025
  5. Detecting Violations of Physical Common Sense in Images: A Challenge Dataset and Effective ModelWeibin Wu, Zitong Wang, Zhengjie Luo, Wenqing Chen et al.ACM MM 2025
Dusk over the sea between two cliffs drawn in fine vertical lines