  While Reinforcement Learning can achieve impressive results for complex
tasks, the learned policies are generally prone to fail in downstream tasks
with even minor model mismatch or unexpected perturbations. Recent works have
demonstrated that a policy population with diverse behavior characteristics can
generalize to downstream environments with various discrepancies. However, such
policies might result in catastrophic damage during the deployment in practical
scenarios like real-world systems due to the unrestricted behaviors of trained
policies. Furthermore, training diverse policies without regulation of the
behavior can result in inadequate feasible policies for extrapolating to a wide
range of test conditions with dynamics shifts. In this work, we aim to train
diverse policies under the regularization of the behavior patterns. We motivate
our paradigm by observing the inverse dynamics in the environment with partial
state information and propose Diversity in Regulation (DiR) training diverse
policies with regulated behaviors to discover desired patterns that benefit the
generalization. Considerable empirical results on various variations of
different environments indicate that our method attains improvements over other
diversity-driven counterparts.
