Lune

CVPR2021顶会

Learning Triadic Belief Dynamics in Nonverbal Communication From Videos

Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, Yixin Zhu

摘要

Humans possess a unique social cognition capability [43, 20] ; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing scene understanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, human poses, and gestures) to represent, model, learn, and infer agents' mental states from pure visual inputs. Crucially, such a mental representation takes the agent's belief into account so that it represents what the true world state is and infers the beliefs in each agent's mental state, which may differ from the true world states. By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents. This "five minds" model differs from prior works that infer beliefs in an infinite recursion; instead, agents' beliefs are converged into a "common mind" [31, 47] . Based on this representation, we further devise a hierarchical energybased model that jointly tracks and predicts all five minds. From this new perspective, a social event is interpreted by a series of nonverbal communication and belief dynamics, which transcends the classic keyframe video summary. In the experiments, we demonstrate that using such a social account provides a better video summary on videos with rich social interactions compared with state-of-the-art keyframe video summary methods.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 4 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • IntentQA: Context-aware Video Intent Reasoning 引用了 Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
  • Read the Room: Video Social Reasoning with Mental-Physical Causal Chains 引用了 IntentQA: Context-aware Video Intent Reasoning
  • Read the Room: Video Social Reasoning with Mental-Physical Causal Chains 引用了 Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper4

  1. IntentQA: Context-aware Video Intent ReasoningJiapeng Li, Ping Wei, Wenjuan Han, Lifeng FanICCV 2023 · 97 次引用
  2. YouRefIt: Embodied Reference Understanding with Language and GestureYixin Chen, Qing Li, Deqian Kong, Yik Lun Kei 等ICCV 2021 · 57 次引用
  3. Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan AcquisitionMatteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi 等ACL 2024
  4. Read the Room: Video Social Reasoning with Mental-Physical Causal ChainsLixing Niu, Jiapeng Li, Xingping Yu, Xinyi Dong 等ICLR 2026

它引用的 Paper2

  1. Gaze360: Physically Unconstrained Gaze Estimation in the WildPetr Kellnhofer, Adrià Recasens, Simon Stent, Wojciech Matusik 等ICCV 2019 · 469 次引用
  2. Understanding Human Gaze Communication by Spatio-Temporal Graph ReasoningLifeng Fan, Wenguan Wang, Song-Chun Zhu, Xinyu Tang 等ICCV 2019 · 124 次引用

相关 Paper

  1. Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction DetectionDongkeun Kim, Minsu Cho, Suha KwakNeurIPS 2025 · 1 次引用
  2. MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze PredictionAnshul Gupta, Samy Tafasca, Arya Farkhondeh, Pierre Vuillecard 等NeurIPS 2024 · 24 次引用
  3. VIBE: Can a VLM Read the Room?Tania Chakraborty, Eylon Caplan, Dan GoldwasserEMNLP Findings 2025 · 2 次引用
  4. PHASE: PHysically-grounded Abstract Social Events for Machine Social PerceptionAviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu 等AAAI 2021 · 44 次引用
  5. Visual Grounding of Learned Physical ModelsYunzhu Li, Toru Lin, Kexin Yi, Daniel Bear 等ICML 2020 · 88 次引用
黄昏的海面,两侧是细线勾勒的悬崖