  In this paper, we define, evaluate, and improve the ``relay-generalization''
performance of reinforcement learning (RL) agents on the out-of-distribution
``controllable'' states. Ideally, an RL agent that generally masters a task
should reach its goal starting from any controllable state of the environment
instead of memorizing a small set of trajectories. For example, a self-driving
system should be able to take over the control from humans in the middle of
driving and must continue to drive the car safely. To practically evaluate this
type of generalization, we start the test agent from the middle of other
independently well-trained \emph{stranger} agents' trajectories. With extensive
experimental evaluation, we show the prevalence of \emph{generalization
failure} on controllable states from stranger agents. For example, in the
Humanoid environment, we observed that a well-trained Proximal Policy
Optimization (PPO) agent, with only 3.9\% failure rate during regular testing,
failed on 81.6\% of the states generated by well-trained stranger PPO agents.
To improve "relay generalization," we propose a novel method called
Self-Trajectory Augmentation (STA), which will reset the environment to the
agent's old states according to the Q function during training. After applying
STA to the Soft Actor Critic's (SAC) training procedure, we reduced the failure
rate of SAC under relay-evaluation by more than three times in most settings
without impacting agent performance and increasing the needed number of
environment interactions. Our code is available at
https://github.com/lan-lc/STA.
