  Off-policy ensemble reinforcement learning (RL) methods have demonstrated
impressive results across a range of RL benchmark tasks. Recent works suggest
that directly imitating experts' policies in a supervised manner before or
during the course of training enables faster policy improvement for an RL
agent. Motivated by these recent insights, we propose Periodic Intra-Ensemble
Knowledge Distillation (PIEKD). PIEKD is a learning framework that uses an
ensemble of policies to act in the environment while periodically sharing
knowledge amongst policies in the ensemble through knowledge distillation. Our
experiments demonstrate that PIEKD improves upon a state-of-the-art RL method
in sample efficiency on several challenging MuJoCo benchmark tasks.
Additionally, we perform ablation studies to better understand PIEKD.
