  Social reasoning necessitates the capacity of theory of mind (ToM), the
ability to contextualise and attribute mental states to others without having
access to their internal cognitive structure. Recent machine learning
approaches to ToM have demonstrated that we can train the observer to read the
past and present behaviours of other agents and infer their beliefs (including
false beliefs about things that no longer exist), goals, intentions and future
actions. The challenges arise when the behavioural space is complex, demanding
skilful space navigation for rapidly changing contexts for an extended period.
We tackle the challenges by equipping the observer with novel neural memory
mechanisms to encode, and hierarchical attention to selectively retrieve
information about others. The memories allow rapid, selective querying of
distal related past behaviours of others to deliberatively reason about their
current mental state, beliefs and future behaviours. This results in ToMMY, a
theory of mind model that learns to reason while making little assumptions
about the underlying mental processes. We also construct a new suite of
experiments to demonstrate that memories facilitate the learning process and
achieve better theory of mind performance, especially for high-demand
false-belief tasks that require inferring through multiple steps of changes.
