  An important difference between brains and deep neural networks is the way
they learn. Nervous systems learn online where a stream of noisy data points
are presented in a non-independent, identically distributed (non-i.i.d.) way.
Further, synaptic plasticity in the brain depends only on information local to
synapses. Deep networks, on the other hand, typically use non-local learning
algorithms and are trained in an offline, non-noisy, i.i.d. setting.
Understanding how neural networks learn under the same constraints as the brain
is an open problem for neuroscience and neuromorphic computing. A standard
approach to this problem has yet to be established. In this paper, we propose
that discrete graphical models that learn via an online maximum a posteriori
learning algorithm could provide such an approach. We implement this kind of
model in a novel neural network called the Sparse Quantized Hopfield Network
(SQHN). We show that SQHNs outperform state-of-the-art neural networks on
associative memory tasks, outperform these models in online, non-i.i.d.
settings, learn efficiently with noisy inputs, and are better than baselines on
a novel episodic memory task.
