  Intrinsic rewards can improve exploration in reinforcement learning, but the
exploration process may suffer from instability caused by non-stationary reward
shaping and strong dependency on hyperparameters. In this work, we introduce
Decoupled RL (DeRL) as a general framework which trains separate policies for
intrinsically-motivated exploration and exploitation. Such decoupling allows
DeRL to leverage the benefits of intrinsic rewards for exploration while
demonstrating improved robustness and sample efficiency. We evaluate DeRL
algorithms in two sparse-reward environments with multiple types of intrinsic
rewards. Our results show that DeRL is more robust to varying scale and rate of
decay of intrinsic rewards and converges to the same evaluation returns than
intrinsically-motivated baselines in fewer interactions. Lastly, we discuss the
challenge of distribution shift and show that divergence constraint
regularisers can successfully minimise instability caused by divergence of
exploration and exploitation policies.
