  To understand cognitive reasoning in the brain, it has been proposed that
symbols and compositions of symbols are represented by activity patterns
(vectors) in a large population of neurons. Formal models implementing this
idea [Plate 2003], [Kanerva 2009], [Gayler 2003], [Eliasmith 2012] include a
reversible superposition operation for representing with a single vector an
entire set of symbols or an ordered sequence of symbols. If the representation
space is high-dimensional, large sets of symbols can be superposed and
individually retrieved. However, crosstalk noise limits the accuracy of
retrieval and information capacity. To understand information processing in the
brain and to design artificial neural systems for cognitive reasoning, a theory
of this superposition operation is essential. Here, such a theory is presented.
The superposition operations in different existing models are mapped to linear
neural networks with unitary recurrent matrices, in which retrieval accuracy
can be analyzed by a single equation. We show that networks representing
information in superposition can achieve a channel capacity of about half a bit
per neuron, a significant fraction of the total available entropy. Going beyond
existing models, superposition operations with recency effects are proposed
that avoid catastrophic forgetting when representing the history of infinite
data streams. These novel models correspond to recurrent networks with
non-unitary matrices or with nonlinear neurons, and can be analyzed and
optimized with an extension of our theory.
