  Transfer reinforcement learning (RL) aims at improving the learning
efficiency of an agent by exploiting knowledge from other source agents trained
on relevant tasks. However, it remains challenging to transfer knowledge
between different environmental dynamics without having access to the source
environments. In this work, we explore a new challenge in transfer RL, where
only a set of source policies collected under diverse unknown dynamics is
available for learning a target task efficiently. To address this problem, the
proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two
key techniques. We learn to aggregate the actions provided by the source
policies adaptively to maximize the target task performance. Meanwhile, we
learn an auxiliary network that predicts residuals around the aggregated
actions, which ensures the target policy's expressiveness even when some of the
source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR
through an extensive experimental evaluation across six simulated environments
ranging from classic control problems to challenging robotics simulations,
under both continuous and discrete action spaces. The demo videos and code are
available on the project webpage: https://omron-sinicx.github.io/multipolar/.
