The Tensor Brain: A Unified Theory of Perception, Memory and Semantic
  Decoding
  We present a unified computational theory of perception and memory. In our
model, perception, episodic memory, and semantic memory are realized by
different functional and operational modes of the oscillating interactions
between an index layer and a representation layer in a bilayer tensor network
(BTN). The memoryless semantic {representation layer} broadcasts information.
In cognitive neuroscience, it would be the "mental canvas", or the "global
workspace" and reflects the cognitive brain state. The symbolic {index layer}
represents concepts and past episodes, whose semantic embeddings are
implemented in the connection weights between both layers. In addition, we
propose a {working memory layer} as a processing center and information buffer.
Episodic and semantic memory realize memory-based reasoning, i.e., the recall
of relevant past information to enrich perception, and are personalized to an
agent's current state, as well as to an agent's unique memories. Episodic
memory stores and retrieves past observations and provides provenance and
context. Recent episodic memory enriches perception by the retrieval of
perceptual experiences, which provide the agent with a sense about the here and
now: to understand its own state, and the world's semantic state in general,
the agent needs to know what happened recently, in recent scenes, and on
recently perceived entities. Remote episodic memory retrieves relevant past
experiences, contributes to our conscious self, and, together with semantic
memory, to a large degree defines who we are as individuals.
