  A critical component to enabling intelligent reasoning in partially
observable environments is memory. Despite this importance, Deep Reinforcement
Learning (DRL) agents have so far used relatively simple memory architectures,
with the main methods to overcome partial observability being either a temporal
convolution over the past k frames or an LSTM layer. More recent work (Oh et
al., 2016) has went beyond these architectures by using memory networks which
can allow more sophisticated addressing schemes over the past k frames. But
even these architectures are unsatisfactory due to the reason that they are
limited to only remembering information from the last k frames. In this paper,
we develop a memory system with an adaptable write operator that is customized
to the sorts of 3D environments that DRL agents typically interact with. This
architecture, called the Neural Map, uses a spatially structured 2D memory
image to learn to store arbitrary information about the environment over long
time lags. We demonstrate empirically that the Neural Map surpasses previous
DRL memories on a set of challenging 2D and 3D maze environments and show that
it is capable of generalizing to environments that were not seen during
training.
