ACL Findings2023
Exploiting Commonsense Knowledge about Objects for Visual Activity Recognition
Abstract
Situation recognition is the task of recognizing the activity depicted in an image, including the people and objects involved. Previous models for this task typically train a classifier to identify the activity using a backbone image feature extractor. We propose that commonsense knowledge about the objects depicted in an image can also be a valuable source of information for activity identification. Previous NLP research has argued that knowledge about the prototypical functions of physical objects is important for language understanding, and NLP techniques have been developed to acquire this knowledge. Our work investigates whether this prototypical function knowledge can also be beneficial for visual situation recognition. We build a framework that incorporates this type of commonsense knowledge in a transformer-based model that is trained to predict the action verb for situation recognition. Our experimental results show that adding prototypical function knowledge about physical objects does improve performance for the visual activity recognition task.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CLIP-Event: Connecting Text and Images with Event Structures cites Learning Transferable Visual Models From Natural Language Supervision
- From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Open-vocabulary Grounded Situation Recognition cites Collaborative Transformers for Grounded Situation Recognition
- From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Open-vocabulary Grounded Situation Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework cites Learning Transferable Visual Models From Natural Language Supervision
- Collaborative Transformers for Grounded Situation Recognition cites Mixture-Kernel Graph Attention Network for Situation Recognition
Cited by top-tier papers2
- From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Open-vocabulary Grounded Situation RecognitionChen Cai, Tianyi Liu, Jianjun Gao, Wenyang Liu et al.ACM MM 2025 · 2 citations
- GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language ModelsDylan Hutson, Daniel Vennemeyer, Aneesh Deshmukh, Justin Zhan et al.EMNLP 2025
Builds on8
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin et al.ICML 2022 · 1,058 citations
- CLIP-Event: Connecting Text and Images with Event StructuresManling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou et al.CVPR 2022 · 103 citations
- Mixture-Kernel Graph Attention Network for Situation RecognitionMohammed Suhail, Leonid SigalICCV 2019 · 32 citations
- Collaborative Transformers for Grounded Situation RecognitionJunhyeong Cho, Youngseok Yoon, Suha KwakCVPR 2022 · 23 citations
Related papers
- Identifying Physical Object Use in SentencesTianyu Jiang, Ellen RiloffEMNLP 2022 · 1 citation
- Learning Prototypical Functions for Physical ArtifactsTianyu Jiang, Ellen RiloffACL 2021
- Understanding 3D Object Interaction from a Single ImageShengyi Qian, David F. FouheyICCV 2023 · 35 citations
- Functionality Discovery and Prediction of Physical ObjectsLei Ji, Botian Shi, Xianglin Guo, Xilin ChenAAAI 2020 · 2 citations
- Joint Commonsense and Relation Reasoning for Image and Video CaptioningJingyi Hou, Xinxiao Wu, Xiaoxun Zhang, Yayun Qi et al.AAAI 2020 · 52 citations
