RobAVA: A Large-Scale Dataset and Baseline Towards Video Based Robotic Arm Action Understanding

Understanding the behaviors of robotic arms is essential for various robotic applications such as logistics management and automated manufacturing. However, the lack of large-scale and diverse datasets significantly hinders progress in video-based robotic arm action understanding.In particular, our RobAVA contains 40k video sequences with video-level fine-grained annotations, covering basic actions such as picking, pushing, and placing, as well as their combinations in different orders and interactions with various objects. Distinguished to existing action recognition benchmarks, RobAVA includes instances of both normal and anomalous executions for each action category. The main challenge in robotic arm action recognition is that a complete action is composed of fundamental, atomic behaviors, requiring models to learn their inter-relationships. To this end, we propose a novel baseline approach, AGPT-Net, which re-defines the problem of understanding robotic arm actions as a task of aligning video sequences with atomic attributes. To enhance AGPT-Net's ability to distinguish normal and anomalous action instances, we introduce a joint semantic space constraint between category and attribute semantics, thereby amplifying the separation between normal and anomalous attribute representations for each action. We conduct extensive experiments to demonstrate AGPT-Net's superiority over other mainstream recognition models.