Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation
Gengcong Yang, Jingyi Zhang, Yong Zhang, Baoyuan Wu, Yujiu Yang
Abstract
To generate "accurate" scene graphs, almost all existing methods predict pairwise relationships in a deterministic manner. However, we argue that visual relationships are often semantically ambiguous. Specifically, inspired by linguistic knowledge, we classify the ambiguity into three types: Synonymy Ambiguity, Hyponymy Ambiguity, and Multi-view Ambiguity. The ambiguity naturally leads to the issue of implicit multi-label, motivating the need for diverse predictions. In this work, we propose a novel plug-andplay Probabilistic Uncertainty Modeling (PUM) module. It models each union region as a Gaussian distribution, whose variance measures the uncertainty of the corresponding visual content. Compared to the conventional deterministic methods, such uncertainty modeling brings stochasticity of feature representation, which naturally enables diverse predictions. As a byproduct, PUM also manages to cover more fine-grained relationships and thus alleviates the issue of bias towards frequent relationships. Extensive experiments on the large-scale Visual Genome benchmark show that combining PUM with newly proposed ResCAGCN can achieve state-of-the-art performances, especially under the mean recall metric. Furthermore, we show the universal effectiveness of PUM by plugging it into some existing models and provide insightful analysis of its ability to generate diverse yet plausible visual relationships.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
21 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- HL-Net: Heterophily Learning Network for Scene Graph Generation cites GPS-Net: Graph Property Sensing Network for Scene Graph Generation
- HL-Net: Heterophily Learning Network for Scene Graph Generation cites Unbiased Scene Graph Generation From Biased Training
- PPDL: Predicate Probability Distribution based Loss for Unbiased Scene Graph Generation cites Unbiased Scene Graph Generation From Biased Training
- Data Uncertainty Learning in Face Recognition cites Robust Person Re-Identification by Modelling Feature Uncertainty
- SGTR: End-to-end Scene Graph Generation with Transformer cites GPS-Net: Graph Property Sensing Network for Scene Graph Generation
- SGTR: End-to-end Scene Graph Generation with Transformer cites Unbiased Scene Graph Generation From Biased Training
- RU-Net: Regularized Unrolling Network for Scene Graph Generation cites GPS-Net: Graph Property Sensing Network for Scene Graph Generation
- RU-Net: Regularized Unrolling Network for Scene Graph Generation cites Unbiased Scene Graph Generation From Biased Training
Cited by top-tier papers21
- SGTR: End-to-end Scene Graph Generation with TransformerRongjie Li, Songyang Zhang, Xuming HeCVPR 2022 · 108 citations
- PPDL: Predicate Probability Distribution based Loss for Unbiased Scene Graph GenerationWei Li, Haiwei Zhang, Qijie Bai, Guoqing Zhao et al.CVPR 2022 · 64 citations
- HL-Net: Heterophily Learning Network for Scene Graph GenerationXin Lin, Changxing Ding, Yibing Zhan, Zijian Li et al.CVPR 2022 · 51 citations
- RU-Net: Regularized Unrolling Network for Scene Graph GenerationXin Lin, Changxing Ding, Jing Zhang, Yibing Zhan et al.CVPR 2022 · 43 citations
- ProbVLM: Probabilistic Adapter for Frozen Vison-Language ModelsUddeshya Upadhyay, Shyamgopal Karthik, Massimiliano Mancini, Zeynep AkataICCV 2023 · 41 citations
Builds on4
- Robust Person Re-Identification by Modelling Feature UncertaintyTianyuan Yu, Da Li, Yongxin Yang, Timothy M. Hospedales et al.ICCV 2019 · 148 citations
- Data Uncertainty Learning in Face RecognitionJie Chang, Zhonghao Lan, Changmao Cheng, Yichen WeiCVPR 2020
- GPS-Net: Graph Property Sensing Network for Scene Graph GenerationXin Lin, Changxing Ding, Jinquan Zeng, Dacheng TaoCVPR 2020
- Unbiased Scene Graph Generation From Biased TrainingKaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi et al.CVPR 2020
Related papers
- Exploiting Relationship for Complex-scene Image GenerationTianyu Hua, Hongdong Zheng, Yalong Bai, Wei Zhang et al.AAAI 2021 · 18 citations
- Not All Relations are Equal: Mining Informative Labels for Scene Graph GenerationArushi Goel, Basura Fernando, Frank Keller, Hakan BilenCVPR 2022 · 30 citations
- Deep Generative Probabilistic Graph Neural Networks for Scene Graph GenerationMahmoud Khademi, Oliver SchulteAAAI 2020 · 55 citations
- Scene Graph Prediction With Limited LabelsRanjay Krishna, Vincent S. Chen, Paroma Varma, Michael S. Bernstein et al.ICCV 2019 · 5 citations
- VrR-VG: Refocusing Visually-Relevant RelationshipsYuanzhi Liang, Yalong Bai, Wei Zhang, Xueming Qian et al.ICCV 2019 · 93 citations
