  Reinforcement learning (RL) has made significant strides in various complex
domains. However, identifying an effective policy via RL often necessitates
extensive exploration. Imitation learning aims to mitigate this issue by using
expert demonstrations to guide exploration. In real-world scenarios, one often
has access to multiple suboptimal black-box experts, rather than a single
optimal oracle. These experts do not universally outperform each other across
all states, presenting a challenge in actively deciding which oracle to use and
in which state. We introduce MAPS and MAPS-SE, a class of policy improvement
algorithms that perform imitation learning from multiple suboptimal oracles. In
particular, MAPS actively selects which of the oracles to imitate and improve
their value function estimates, and MAPS-SE additionally leverages an active
state exploration criterion to determine which states one should explore. We
provide a comprehensive theoretical analysis and demonstrate that MAPS and
MAPS-SE enjoy sample efficiency advantage over the state-of-the-art policy
improvement algorithms. Empirical results show that MAPS-SE significantly
accelerates policy optimization via state-wise imitation learning from multiple
oracles across a broad spectrum of control tasks in the DeepMind Control Suite.
Our code is publicly available at: https://github.com/ripl/maps.
