Learning Triadic Belief Dynamics in Nonverbal Communication From Videos
Lifeng Fan, Shuwen Qiu, Zilong Zheng, Tao Gao, Song-Chun Zhu, Yixin Zhu
摘要
Humans possess a unique social cognition capability [43, 20] ; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing scene understanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, human poses, and gestures) to represent, model, learn, and infer agents' mental states from pure visual inputs. Crucially, such a mental representation takes the agent's belief into account so that it represents what the true world state is and infers the beliefs in each agent's mental state, which may differ from the true world states. By aggregating different beliefs and true world states, our model essentially forms "five minds" during the interactions between two agents. This "five minds" model differs from prior works that infer beliefs in an infinite recursion; instead, agents' beliefs are converged into a "common mind" [31, 47] . Based on this representation, we further devise a hierarchical energybased model that jointly tracks and predicts all five minds. From this new perspective, a social event is interpreted by a series of nonverbal communication and belief dynamics, which transcends the classic keyframe video summary. In the experiments, we demonstrate that using such a social account provides a better video summary on videos with rich social interactions compared with state-of-the-art keyframe video summary methods.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 4 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- IntentQA: Context-aware Video Intent Reasoning 引用了 Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
- Read the Room: Video Social Reasoning with Mental-Physical Causal Chains 引用了 IntentQA: Context-aware Video Intent Reasoning
- Read the Room: Video Social Reasoning with Mental-Physical Causal Chains 引用了 Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning
引用它的顶会 Paper4
- IntentQA: Context-aware Video Intent ReasoningJiapeng Li, Ping Wei, Wenjuan Han, Lifeng FanICCV 2023 · 97 次引用
- YouRefIt: Embodied Reference Understanding with Language and GestureYixin Chen, Qing Li, Deqian Kong, Yik Lun Kei 等ICCV 2021 · 57 次引用
- Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan AcquisitionMatteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi 等ACL 2024
- Read the Room: Video Social Reasoning with Mental-Physical Causal ChainsLixing Niu, Jiapeng Li, Xingping Yu, Xinyi Dong 等ICLR 2026
它引用的 Paper2
相关 Paper
- Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction DetectionDongkeun Kim, Minsu Cho, Suha KwakNeurIPS 2025 · 1 次引用
- MTGS: A Novel Framework for Multi-Person Temporal Gaze Following and Social Gaze PredictionAnshul Gupta, Samy Tafasca, Arya Farkhondeh, Pierre Vuillecard 等NeurIPS 2024 · 24 次引用
- VIBE: Can a VLM Read the Room?Tania Chakraborty, Eylon Caplan, Dan GoldwasserEMNLP Findings 2025 · 2 次引用
- PHASE: PHysically-grounded Abstract Social Events for Machine Social PerceptionAviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu 等AAAI 2021 · 44 次引用
- Visual Grounding of Learned Physical ModelsYunzhu Li, Toru Lin, Kexin Yi, Daniel Bear 等ICML 2020 · 88 次引用
