  Effective exploration is critical for reinforcement learning agents in
environments with sparse rewards or high-dimensional state-action spaces.
Recent works based on state-visitation counts, curiosity and
entropy-maximization generate intrinsic reward signals to motivate the agent to
visit novel states for exploration. However, the agent can get distracted by
perturbations to sensor inputs that contain novel but task-irrelevant
information, e.g. due to sensor noise or changing background. In this work, we
introduce the sequential information bottleneck objective for learning
compressed and temporally coherent representations by modelling and compressing
sequential predictive information in time-series observations. For efficient
exploration in noisy environments, we further construct intrinsic rewards that
capture task-relevant state novelty based on the learned representations. We
derive a variational upper bound of our sequential information bottleneck
objective for practical optimization and provide an information-theoretic
interpretation of the derived upper bound. Our experiments on a set of
challenging image-based simulated control tasks show that our method achieves
better sample efficiency, and robustness to both white noise and natural video
backgrounds compared to state-of-art methods based on curiosity, entropy
maximization and information-gain.
