Poly-Autoregressive Prediction for Modeling Interactions

Figure 1. Inference for (a) autoregressive (AR) models and (b) our proposed poly-autoregressive (PAR) model. Solid indicates ground-truth tokens which represent a tracked data modality such as action or 6DOF pose; striped represents predicted output tokens. Color denotes agent identity. Compared to AR models, the PAR model takes other agents' tokens as inputs when making a prediction for the next timestep.