  Go-Explore achieved breakthrough performance on challenging reinforcement
learning (RL) tasks with sparse rewards. The key insight of Go-Explore was that
successful exploration requires an agent to first return to an interesting
state ('Go'), and only then explore into unknown terrain ('Explore'). We refer
to such exploration after a goal is reached as 'post-exploration'. In this
paper, we present a clear ablation study of post-exploration in a general
intrinsically motivated goal exploration process (IMGEP) framework, that the
Go-Explore paper did not show. We study the isolated potential of
post-exploration, by turning it on and off within the same algorithm under both
tabular and deep RL settings on both discrete navigation and continuous control
tasks. Experiments on a range of MiniGrid and Mujoco environments show that
post-exploration indeed helps IMGEP agents reach more diverse states and boosts
their performance. In short, our work suggests that RL researchers should
consider to use post-exploration in IMGEP when possible since it is effective,
method-agnostic and easy to implement.
