  Intelligence necessitates memory. Without memory, humans fail to perform
various nontrivial tasks such as reading novels, playing games or solving
maths. As the ultimate goal of machine learning is to derive intelligent
systems that learn and act automatically just like human, memory construction
for machine is inevitable. Artificial neural networks model neurons and
synapses in the brain by interconnecting computational units via weights, which
is a typical class of machine learning algorithms that resembles memory
structure. Their descendants with more complicated modeling techniques (a.k.a
deep learning) have been successfully applied to many practical problems and
demonstrated the importance of memory in the learning process of machinery
systems. Recent progresses on modeling memory in deep learning have revolved
around external memory constructions, which are highly inspired by
computational Turing models and biological neuronal systems. Attention
mechanisms are derived to support acquisition and retention operations on the
external memory. Despite the lack of theoretical foundations, these approaches
have shown promises to help machinery systems reach a higher level of
intelligence. The aim of this thesis is to advance the understanding on memory
and attention in deep learning. Its contributions include: (i) presenting a
collection of taxonomies for memory, (ii) constructing new memory-augmented
neural networks (MANNs) that support multiple control and memory units, (iii)
introducing variability via memory in sequential generative models, (iv)
searching for optimal writing operations to maximise the memorisation capacity
in slot-based memory networks, and (v) simulating the Universal Turing Machine
via Neural Stored-program Memory-a new kind of external memory for neural
networks.
