trajdata: A Unified Interface to Multiple Human Trajectory Datasets

The field of trajectory forecasting has grown significantly in recent years, partially owing to the release of numerous large-scale, real-world human trajectory datasets for autonomous vehicles (AVs) and pedestrian motion tracking. While such datasets have been a boon for the community, they each use custom and unique data formats and APIs, making it cumbersome for researchers to train and evaluate methods across multiple datasets. To remedy this, we present trajdata: a unified interface to multiple human trajectory datasets. At its core, trajdata provides a simple, uniform, and efficient representation and API for trajectory and map data. As a demonstration of its capabilities, in this work we conduct a comprehensive empirical evaluation of existing trajectory datasets, providing users with a rich understanding of the data underpinning much of current pedestrian and AV motion forecasting research, and proposing suggestions for future datasets from these insights. trajdata is permissively licensed (Apache 2.0) and can be accessed online at https://github.com/NVlabs/trajdata . Introduction Research in trajectory forecasting (i.e., predicting where an agent will be in the future) has grown significantly in recent years, partially owing to the success of deep learning methods on the task [1]; availability of new large-scale, real-world datasets (see Fig. 1 ); and investment in its deployment within domains such as autonomous vehicles (AVs) [2, 3, 4, 5, 6, 7, 8, 9] and social robots [10, 11, 12] . In addition, recent dataset releases have held associated prediction challenges which have periodically benchmarked the field and spurned new developments [13, 14, 15, 16] . While this has been a boon for research progress, each dataset has a unique data format and development API, making it cumbersome for researchers to train and evaluate methods across multiple datasets. For instance, the recent Waymo Open Motion dataset employs binary TFRecords [17] which differ significantly from nuScenes' foreign-key format [18] and Woven Planet (Lyft) Level 5's compressed zarr files [19] . The variety of data formats has also hindered research on topics which either require or greatly benefit from multi-dataset comparisons, such as prediction model generalization (e.g., [20, 21] ). To remedy this, we present trajdata: a unified interface to multiple human trajectory datasets. Contributions. Our key contributions are threefold. First, we introduce a standard and simple data format for trajectory and map data, as well as an extensible API to access and transform such data for research use. Second, we conduct a comprehensive empirical evaluation of existing trajectory datasets, providing users with a richer understanding of the data underpinning much of pedestrian and AV motion forecasting research. Finally, we leverage insights from these analyses to provide suggestions for future dataset releases. Preprint. Under review.