  We present a new computing model for intrinsic rewards in reinforcement
learning that addresses the limitations of existing surprise-driven
explorations. The reward is the novelty of the surprise rather than the
surprise norm. We estimate the surprise novelty as retrieval errors of a memory
network wherein the memory stores and reconstructs surprises. Our surprise
memory (SM) augments the capability of surprise-based intrinsic motivators,
maintaining the agent's interest in exciting exploration while reducing
unwanted attraction to unpredictable or noisy observations. Our experiments
demonstrate that the SM combined with various surprise predictors exhibits
efficient exploring behaviors and significantly boosts the final performance in
sparse reward environments, including Noisy-TV, navigation and challenging
Atari games.
