  Efficient exploration is a challenging topic in reinforcement learning,
especially for sparse reward tasks. To deal with the reward sparsity, people
commonly apply intrinsic rewards to motivate agents to explore the state space
efficiently. In this paper, we introduce a new intrinsic reward design called
GoBI - Go Beyond Imagination, which combines the traditional lifelong novelty
motivation with an episodic intrinsic reward that is designed to maximize the
stepwise reachability expansion. More specifically, we apply learned world
models to generate predicted future states with random actions. States with
more unique predictions that are not in episodic memory are assigned high
intrinsic rewards. Our method greatly outperforms previous state-of-the-art
methods on 12 of the most challenging Minigrid navigation tasks and improves
the sample efficiency on locomotion tasks from DeepMind Control Suite.
