Context-Enhanced Memory-Refined Transformer for Online Action Detection

Online Action Detection (OAD) detects actions in streaming videos using past observations. State-of-the-art OAD approaches model past observations and their interactions with an anticipated future. The past is encoded using shortand long-term memories to capture immediate and longrange dependencies, while anticipation compensates for missing future context. We identify a training-inference discrepancy in existing OAD methods that hinders learning effectiveness. The training uses varying lengths of shortterm memory, while inference relies on a full-length shortterm memory. As a remedy, we propose a Context-enhanced Memory-Refined Transformer (CMeRT). CMeRT introduces a context-enhanced encoder to improve frame representations using additional near-past context. It also features a memory-refined decoder to leverage near-future generation to enhance performance. CMeRT 1 achieves state-of-theart in online detection and anticipation on THUMOS'14, CrossTask, and EPIC-Kitchens-100.