  Transferring reinforcement learning policies trained in physics simulation to
the real hardware remains a challenge, known as the "sim-to-real" gap. Domain
randomization is a simple yet effective technique to address dynamics
discrepancies across source and target domains, but its success generally
depends on heuristics and trial-and-error. In this work we investigate the
impact of randomized parameter selection on policy transferability across
different types of domain discrepancies. Contrary to common practice in which
kinematic parameters are carefully measured while dynamic parameters are
randomized, we found that virtually randomizing kinematic parameters (e.g.,
link lengths) during training in simulation generally outperforms dynamic
randomization. Based on this finding, we introduce a new domain adaptation
algorithm that utilizes simulated kinematic parameters variation. Our
algorithm, Multi-Policy Bayesian Optimization, trains an ensemble of universal
policies conditioned on virtual kinematic parameters and efficiently adapts to
the target environment using a limited number of target domain rollouts. We
showcase our findings on a simulated quadruped robot in five different target
environments covering different aspects of domain discrepancies.
