  A robot operating in a household makes observations of multiple objects as it
moves around over the course of days or weeks. The objects may be moved by
inhabitants, but not completely at random. The robot may be called upon later
to retrieve objects and will need a long-term object-based memory in order to
know how to find them. Existing work in semantic slam does not attempt to
capture the dynamics of object movement. In this paper, we combine some aspects
of classic techniques for data-association filtering with modern
attention-based neural networks to construct object-based memory systems that
operate on high-dimensional observations and hypotheses. We perform end-to-end
learning on labeled observation trajectories to learn both the transition and
observation models. We demonstrate the system's effectiveness in maintaining
memory of dynamically changing objects in both simulated environment and real
images, and demonstrate improvements over classical structured approaches as
well as unstructured neural approaches. Additional information available at
project website: https://yilundu.github.io/obm/.
