Hierarchical Graph Attention Network for Visual Relationship Detection

Visual Relationship Detection (VRD) aims to describe the relationship between two objects by providing a structural triplet shown as <subject-predicate-object>. Existing graph-based methods mainly represent the relationships by an object-level graph, which ignores to model the tripletlevel dependencies. In this work, a Hierarchical Graph Attention Network (HGAT) is proposed to capture the dependencies on both object-level and triplet-level. Objectlevel graph aims to capture the interactions between objects, while the triplet-level graph models the dependencies among relation triplets. In addition, prior knowledge and attention mechanism are introduced to fix the redundant or missing edges on graphs that are constructed according to spatial correlation. With these approaches, nodes are allowed to attend over their spatial and semantic neighborhoods' features based on the visual or semantic feature correlation. Experimental results on the well-known VG and VRD datasets demonstrate that our model significantly outperforms the state-of-the-art methods.