  Recent research developing neural network architectures with external memory
have often used the benchmark bAbI question and answering dataset which
provides a challenging number of tasks requiring reasoning. Here we employed a
classic associative inference task from the memory-based reasoning neuroscience
literature in order to more carefully probe the reasoning capacity of existing
memory-augmented architectures. This task is thought to capture the essence of
reasoning -- the appreciation of distant relationships among elements
distributed across multiple facts or memories. Surprisingly, we found that
current architectures struggle to reason over long distance associations.
Similar results were obtained on a more complex task involving finding the
shortest path between nodes in a path. We therefore developed MEMO, an
architecture endowed with the capacity to reason over longer distances. This
was accomplished with the addition of two novel components. First, it
introduces a separation between memories (facts) stored in external memory and
the items that comprise these facts in external memory. Second, it makes use of
an adaptive retrieval mechanism, allowing a variable number of "memory hops"
before the answer is produced. MEMO is capable of solving our novel reasoning
tasks, as well as match state of the art results in bAbI.
