Dynamic Kernel Distillation for Efficient Pose Estimation in Videos

Existing video-based human pose estimation methods extensively apply large networks onto every frame in the video to localize body joints, which suffer high computational cost and hardly meet the low-latency requirement in realistic applications. To address this issue, we propose a novel Dynamic Kernel Distillation (DKD) model to facilitate small networks for estimating human poses in videos, thus significantly lifting the efficiency. In particular, DKD introduces a light-weight distillator to online distill pose kernels via leveraging temporal cues from the previous frame in a one-shot feed-forward manner. Then, DKD simplifies body joint localization into a matching procedure between the pose kernels and the current frame, which can be efficiently computed via simple convolution. In this way, DKD fast transfers pose knowledge from one frame to provide compact guidance for body joint localization in the following frame, which enables utilization of small networks in video-based pose estimation. To facilitate the training process, DKD exploits a temporally adversarial training strategy that introduces a temporal discriminator to help generate temporally coherent pose kernels and pose estimation results within a long range. Experiments on Penn Action and Sub-JHMDB benchmarks demonstrate outperforming efficiency of DKD, specifically, 10× flops reduction and 2× speedup over previous best model, and its state-of-the-art accuracy. * This work was partly done while Xuecheng was an intern as Snap Inc. Small CNN Pose Kernel Distillator Matching Frame t-1 Frame t Small CNN Matching Frame t+1 Small CNN Pose Kernel Distillator Matching (a) Our DKD Model RNN or Optical Flow Large CNN Classification Frame t-1 RNN or Optical Flow Large CNN Frame t Large CNN Frame t+1 Classification Classification (b) The Traditional Model