Multimodal Global Relation Knowledge Distillation for Egocentric Action Anticipation

In this paper, we consider the task of action anticipation on egocentric videos. Previous methods ignore explicit modeling of the global context relation among past and future actions, which is not an easy task due to the vacancy of unobserved videos. To solve this problem, we propose a Multimodal Global Relation Knowledge Distillation (MGRKD) framework to distill the knowledge learned from full videos to improve the action anticipation task on partially observed videos. The proposed MGRKD has a teacher-student learning strategy, where either the teacher or student model has three branches of global relation graph networks (GRGN) to explore the pairwise relations between past and future actions based on three kinds of features (i.e., RGB, motion or object). The teacher model has a similar architecture with the student model, except that the teacher model uses true feature of the future video snippet to build the graph in GRGN while the student model uses a progressive GRU to predict an initialized node feature of future snippet in GRGN. Through the teacher-student learning strategy, the discriminative features and relation knowledge of the past and future actions learned in the teacher model can be distilled to the student model. The experiments on two egocentric video datasets EPIC-Kitchens and EGTEA Gaze+ show that the proposed framework achieves state-of-the-art performances.