Representation Learning via Global Temporal Alignment and Cycle-Consistency

We introduce a weakly supervised method for representation learning based on aligning temporal sequences (e.g., videos) of the same process (e.g., human action). The main idea is to use the global temporal ordering of latent correspondences across sequence pairs as a supervisory signal. In particular, we propose a loss based on scoring the optimal sequence alignment to train an embedding network. Our loss is based on a novel probabilistic path finding view of dynamic time warping (DTW) that contains the following three key features: (i) the local path routing decisions are contrastive and differentiable, (ii) pairwise distances are cast as probabilities that are contrastive as well, and (iii) our formulation naturally admits a global cycleconsistency loss that verifies correspondences. For evaluation, we consider the tasks of fine-grained action classification, few shot learning, and video synchronization. We report significant performance increases over previous methods. In addition, we report two applications of our temporal alignment framework, namely 3D pose reconstruction and fine-grained audio/visual retrieval.

capable of scaling up to large amounts of data yet supporting finer-grained video understanding.

As outlined in Fig. 1, given a set of paired sequences capturing the same process (e.g., tennis forehand) but highly varied (i.e., different participants, action executions, scenes, and camera viewpoints), our method trains an embedding network to support the recovery of their latent temporal alignment. We refer to our method as weakly supervised, as only readily available sequence-level labels (e.g., tennis forehand) are required to construct training pairings containing the same process. Given such a pair of sequences, we use their latent temporal alignment as a supervisory signal to learn fine-grained temporal distinctions.

Key to our proposed method is a novel dynamic time warping (DTW) formulation to score global alignments between paired sequences. DTW enforces a stronger constraint over simply considering local (soft) nearest neighbour correspondences [15], since the temporal ordering of the matches in the sequences are taken into account. We depart from previous differentiable DTW methods [30,8,6]