Deep Generative Probabilistic Graph Neural Networks for Scene Graph Generation

We propose a new algorithm, called Deep Generative Probabilistic Graph Neural Networks (DG-PGNN), to generate a scene graph for an image. The input to DG-PGNN is an image, together with a set of region-grounded captions and object bounding-box proposals for the image. To generate the scene graph, DG-PGNN constructs and updates a new model, called a Probabilistic Graph Network (PGN). A PGN can be thought of a scene graph with uncertainty: it represents each node and each edge by a CNN feature vector and defines a probability mass function (PMF) for node-type (object category) of each node and edge-type (predicate class) of each edge. The DG-PGNN sequentially adds a new node to the current PGN by learning the optimal ordering in a Deep Q-learning framework, where states are partial PGNs, actions choose a new node, and rewards are defined based on the groundtruth. After adding a node, DG-PGNN uses message passing to update the feature vectors of the current PGN by leveraging contextual relationship information, object co-occurrences, and language priors from captions. The updated features are then used to fine-tune the PMFs. Our experiments show that the proposed algorithm significantly outperforms the state-of-the-art results on the Visual Genome dataset. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. Recently, [4] proposed a model to detect a set of relationships using language priors from semantic word embeddings. [11] proposed to learn a contextualized representation for nodes and edges by assuming a fixed linear ordering for the bounding-boxes and applying a bidirectional LSTM. In [7] , authors proposed to train a CNN which takes in an image and produces a scene graph in an end-to-end framework. In [8] , authors introduced a model which composes dynamic tree structures that put the objects in an image into a visual context to improve scene graph generation accuracy. In [9] , authors developed a model for scene graph generation which uses an RNN and learns to improves its predictions iteratively through message passing across the scene graph. Proposed Algorithm A scene graph for an image is generated from a Probabilistic Graph Network (PGN). We introduce the PGN, then explain the DG-PGNN algorithm for generating a PGN for an image.