BABEL: Bodies, Action and Behavior With English Labels

Understanding the semantics of human movement -the what, how and why of the movement -is an important problem that requires datasets of human actions with semantic labels. Existing datasets take one of two approaches. Large-scale video datasets contain many action labels but do not contain ground-truth 3D human motion. Alternatively, motion-capture (mocap) datasets have precise body motions but are limited to a small number of actions. To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences. BABEL consists of language labels for over 43 hours of mocap sequences from AMASS, containing over 250 unique actions. Each action label in BABEL is precisely aligned with the duration of the corresponding action in the mocap sequence. BABELalso allows overlap of multiple actions, that may each span different durations. This results in a total of over 66000 action segments. The dense annotations can be leveraged for tasks like action recognition, temporal localization, motion synthesis, etc. To demonstrate the value of BABEL as a benchmark, we evaluate the performance of models on 3D action recognition. We demonstrate that BABEL poses interesting learning challenges that are applicable to real-world scenarios, and can serve as a useful benchmark for progress in 3D action recognition. The dataset, baseline methods, and evaluation code are available and supported for academic research purposes at https://babel.is.tue.mpg.de/ . bob head, head bang, nod head, nod, move head side ways, tilt head back, move head around, twist head, tilt head to the right, move head in a circle pace quickly, walk away, runway walk, power walk, wobble walk, slow walk back and forth, speed walk, sidle, strolls, catwalk, walking backwards, walking in circles, zigzag, stride, saunter, plod, shamble, strut, wander, hobble, tread, swagger dance with partner, ballet dancing, interpretive dancing, ginga dance, doing a nutty dance, expression dance, slow dancing with imaginary partner, teacup dance, waltz, fish flop, plie, shimmy, sway scratch hands, scratch chin, scratch arm, scratch face, scratch head, scratch nose, scratching side, scratch waist with right hand, itch butt, itch leg lean backwards to the left, lean to left, lean right, lean left knee forward, lean against object, tilt right place object on upper shelf, put object on table, place ice in glass, put book on shelf, place drink, put box on shelf, place object on ground, place item in waist band, loading stuff into a truck twist wrists in a circle, twist body to the left, twist body to the right, jump twist, whirl body, twirl leg, twist left leg, twist head, twist cork, twirl, barrel roll