  Efficient exploration in reinforcement learning is a challenging problem
commonly addressed through intrinsic rewards. Recent prominent approaches are
based on state novelty or variants of artificial curiosity. However, directly
applying them to partially observable environments can be ineffective and lead
to premature dissipation of intrinsic rewards. Here we propose random curiosity
with general value functions (RC-GVF), a novel intrinsic reward function that
draws upon connections between these distinct approaches. Instead of using only
the current observation's novelty or a curiosity bonus for failing to predict
precise environment dynamics, RC-GVF derives intrinsic rewards through
predicting temporally extended general value functions. We demonstrate that
this improves exploration in a hard-exploration diabolical lock problem.
Furthermore, RC-GVF significantly outperforms previous methods in the absence
of ground-truth episodic counts in the partially observable MiniGrid
environments. Panoramic observations on MiniGrid further boost RC-GVF's
performance such that it is competitive to baselines exploiting privileged
information in form of episodic counts.
