Part-Aware Bottom-Up Group Reasoning for Fine-Grained Social Interaction Detection

Social interactions often emerge from subtle, fine-grained cues such as facial expressions, gaze, and gestures. However, existing methods for social interaction detection overlook such nuanced cues and primarily rely on holistic representations of individuals. Moreover, they directly detect social groups without explicitly modeling the underlying interactions between individuals. These drawbacks limit their ability to capture localized social signals and introduce ambiguity when group configurations should be inferred from social interactions grounded in nuanced cues. In this work, we propose a part-aware bottom-up group reasoning framework for fine-grained social interaction detection. The proposed method infers social groups and their interactions using body part features and their interpersonal relations. Our model first detects individuals and enhances their features using part-aware cues, and then infers group configuration by associating individuals via similarity-based reasoning, which considers not only spatial relations but also subtle social cues that signal interactions, leading to more accurate group inference. Experiments on the NVI dataset demonstrate that our method outperforms prior methods, achieving the new state of the art, while additional results on the Café dataset further validate its generalizability to group activity understanding.

Recently, a new task of detecting multi-person interactions based on such fine-grained, nuanced, and ambiguous social cues has been introduced along with a dedicated dataset, NVI [51]. The task, named nonverbal interaction detection (NVI-DET), is formulated to detect each individual, identify the social group they belong to, and classify their fine-grained social interaction, encapsulated as a triplet <individual, group, interaction>. This formulation draws inspiration from human-object interaction (HOI) detection [31,44,58], but differs in that it targets human-human social interactions, including facial expression, gesture, posture, gaze, and touch, that are inherently more nuanced and 39th Conference on Neural Information Processing Systems (NeurIPS 2025).