DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding
Xiaoyu Guo, Yuan-Fang Li, Gholamreza Haffari
Abstract
Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. One representative benchmark for its study is Social Intelligence Queries (Social-IQ), a dataset of multiple-choice questions on videos of complex social interactions. We define a comprehensive methodology to study the soundness of Social-IQ, as the soundness of such benchmark datasets is crucial to the investigation of the underlying research problem. Our analysis reveals that Social-IQ contains substantial biases, which can be exploited by a moderately strong language model to learn spurious correlations to achieve perfect performance without being given the context or even the question. We introduce DeSIQ, a new challenging dataset, constructed by applying simple perturbations to Social-IQ. Our empirical analysis shows DeSIQ significantly reduces the biases in the original Social-IQ dataset. Furthermore, we examine and shed light on the effect of model size, model style, learning settings, commonsense knowledge, and multi-modality on the new benchmark performance. Our new dataset, observations and findings open up important research questions for the study of social intelligence.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers3
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of MindKazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Saki Mizuno et al.AAAI 2025 · 10 citations
- Mitigating Language Bias of LMMs in Social Intelligence Understanding with Virtual Counterfactual CalibrationPeng Chen, Xiao-Yu Guo, Yuan-Fang Li, Xiaowang Zhang et al.EMNLP 2024 · 1 citation
- Social Genome: Grounded Social Reasoning Abilities of Multimodal ModelsLeena Mathur, Marian Qian, Paul Pu Liang, Louis-Philippe MorencyEMNLP 2025
Builds on10
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
- Aligning AI With Shared Human ValuesDan Hendrycks, Collin Burns, Steven Basart, Andrew Critch et al.ICLR 2021 · 878 citations
- TVQA+: Spatio-Temporal Grounding for Video Question AnsweringJie Lei, Licheng Yu, Tamara L. Berg, Mohit BansalACL 2020 · 173 citations
Related papers
- SIV-Bench: A Video Benchmark for Social Interaction Understanding and ReasoningFanqi Kong, Weiqin Zu, Xinyu Chen, Yaodong Yang et al.ACL Findings 2026
- BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of MindYuanyuan Mao, Xin Lin, Qin Ni, Liang HeAAAI 2024 · 6 citations
- When and Why Does Bias Mitigation Work?Abhilasha Ravichander, Joe Stacey, Marek ReiEMNLP Findings 2023 · 1 citation
- COM2SENSE: A Commonsense Reasoning Benchmark with Complementary SentencesShikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi et al.ACL Findings 2021 · 30 citations
- What's Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLMsJinhao Pan, Chahat Raj, Ziyu Yao, Ziwei ZhuEMNLP Findings 2025
