Optimizing Probabilistic Box Embeddings with Distance Measures

Recently, geometric-inspired embedding methods draw research interests for their superior ability in representing transitive and asymmetric relations. A typical example, box embeddings, in which objects are parameterized as axis-aligned hyper-rectangles (i.e. boxes), can effectively model the partial orders and similarities between objects with the inclusion and overlapping relations of the boxes. However, the hard edges of the boxes present difficulties for gradient-based optimization. In this paper, we first identify two problems that may hinder the optimization of box embeddings, namely the zero-gradient problem and the vanishing gradient problem in high dimensional settings. Then, we propose a simple yet effective framework (Box++) to tackle these problems. For the first problem, Box++ combines the overlapping volumes and distance measures in optimizing box embeddings. The distance measures can naturally measure the “degree of disjointedness” for disjoint boxes and provide reasonable gradients for optimization. For the second problem, we theoretically prove that under certain conditions, the gradient would vanish exponentially, and therefore, make the optimization converges to suboptimal solutions. We further design a gradient normalization strategy for the Box++ models to alleviate this problem by rescaling the vanishing gradient. Extensive experiments on real-world datasets demonstrate the effectiveness of the proposed framework.