  In environments with sparse rewards, finding a good inductive bias for
exploration is crucial to the agent's success. However, there are two competing
goals: novelty search and systematic exploration. While existing approaches
such as curiosity-driven exploration find novelty, they sometimes do not
systematically explore the whole state space, akin to depth-first-search vs
breadth-first-search. In this paper, we propose a new intrinsic reward that is
cyclophobic, i.e., it does not reward novelty, but punishes redundancy by
avoiding cycles. Augmenting the cyclophobic intrinsic reward with a sequence of
hierarchical representations based on the agent's cropped observations we are
able to achieve excellent results in the MiniGrid and MiniHack environments.
Both are particularly hard, as they require complex interactions with different
objects in order to be solved. Detailed comparisons with previous approaches
and thorough ablation studies show that our newly proposed cyclophobic
reinforcement learning is more sample efficient than other state of the art
methods in a variety of tasks.
