Cross-Episodic Curriculum for Transformer Agents

We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings, and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is opensourced on the project website cec-agent.github.io to facilitate research on Transformer agent learning. 1 Following the canonical definition in Sutton and Barto [73] , we refer to the sequences of agent-environment interaction with clearly identified initial and terminal states as "episodes". We interchangeably use "episode", "trial", and "trajectory" in this work. 37th Conference on Neural Information Processing Systems (NeurIPS 2023).