Lune

CVPR2021Top-tier venue

Learning Triadic Belief Dynamics in Nonverbal Communication From Videos

Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, Yixin Zhu

Abstract

Humans possess a unique social cognition capability [43, 20] ; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing scene understanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, human poses, and gestures) to represent, model, learn, and infer agents' mental states from pure visual inputs. Crucially, such a mental representation takes the agent's belief into account so that it represents what the true world state is and infers the beliefs in each agent's mental state, which may differ from the true world states. By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents. This "five minds" model differs from prior works that infer beliefs in an infinite recursion; instead, agents' beliefs are converged into a "common mind" [31, 47] . Based on this representation, we further devise a hierarchical energybased model that jointly tracks and predicts all five minds. From this new perspective, a social event is interpreted by a series of nonverbal communication and belief dynamics, which transcends the classic keyframe video summary. In the experiments, we demonstrate that using such a social account provides a better video summary on videos with rich social interactions compared with state-of-the-art keyframe video summary methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • IntentQA: Context-aware Video Intent Reasoning cites Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
  • Read the Room: Video Social Reasoning with Mental-Physical Causal Chains cites IntentQA: Context-aware Video Intent Reasoning
  • Read the Room: Video Social Reasoning with Mental-Physical Causal Chains cites Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. IntentQA: Context-aware Video Intent ReasoningJiapeng Li, Ping Wei, Wenjuan Han, Lifeng FanICCV 2023 · 97 citations
  2. YouRefIt: Embodied Reference Understanding with Language and GestureYixin Chen, Qing Li, Deqian Kong, Yik Lun Kei et al.ICCV 2021 · 57 citations
  3. Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan AcquisitionMatteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi et al.ACL 2024
  4. Read the Room: Video Social Reasoning with Mental-Physical Causal ChainsLixing Niu, Jiapeng Li, Xingping Yu, Xinyi Dong et al.ICLR 2026

Builds on2

  1. Gaze360: Physically Unconstrained Gaze Estimation in the WildPetr Kellnhofer, Adrià Recasens, Simon Stent, Wojciech Matusik et al.ICCV 2019 · 469 citations
  2. Understanding Human Gaze Communication by Spatio-Temporal Graph ReasoningLifeng Fan, Wenguan Wang, Song-Chun Zhu, Xinyu Tang et al.ICCV 2019 · 124 citations

Related papers

  1. Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction DetectionDongkeun Kim, Minsu Cho, Suha KwakNeurIPS 2025 · 1 citation
  2. MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze PredictionAnshul Gupta, Samy Tafasca, Arya Farkhondeh, Pierre Vuillecard et al.NeurIPS 2024 · 24 citations
  3. VIBE: Can a VLM Read the Room?Tania Chakraborty, Eylon Caplan, Dan GoldwasserEMNLP Findings 2025 · 2 citations
  4. PHASE: PHysically-grounded Abstract Social Events for Machine Social PerceptionAviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu et al.AAAI 2021 · 44 citations
  5. Visual Grounding of Learned Physical ModelsYunzhu Li, Toru Lin, Kexin Yi, Daniel Bear et al.ICML 2020 · 88 citations
Dusk over the sea between two cliffs drawn in fine vertical lines