  We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the
learning efficiency and generalization of Transformer agents. Central to CEC is
the placement of cross-episodic experiences into a Transformer's context, which
forms the basis of a curriculum. By sequentially structuring online learning
trials and mixed-quality demonstrations, CEC constructs curricula that
encapsulate learning progression and proficiency increase across episodes. Such
synergy combined with the potent pattern recognition capabilities of
Transformer models delivers a powerful cross-episodic attention mechanism. The
effectiveness of CEC is demonstrated under two representative scenarios: one
involving multi-task reinforcement learning with discrete control, such as in
DeepMind Lab, where the curriculum captures the learning progression in both
individual and progressively complex settings; and the other involving
imitation learning with mixed-quality data for continuous control, as seen in
RoboMimic, where the curriculum captures the improvement in demonstrators'
expertise. In all instances, policies resulting from CEC exhibit superior
performance and strong generalization. Code is open-sourced at
https://cec-agent.github.io/ to facilitate research on Transformer agent
learning.
