  The neural implementation of operant conditioning with few trials is unclear.
We propose a Hippocampus-Inspired Cognitive Architecture (HICA) as a neural
mechanism for operant conditioning. HICA explains a learning mechanism in which
agents can learn a new behavior policy in a few trials, as mammals do in
operant conditioning experiments. HICA is composed of two different types of
modules. One is a universal learning module type that represents a cortical
column in the neocortex gray matter. The working principle is modeled as
Modulated Heterarchical Prediction Memory (mHPM). In mHPM, each module learns
to predict a succeeding input vector given the sequence of the input vectors
from lower layers and the context vectors from higher layers. The prediction is
fed into the lower layers as a context signal (top-down feedback signaling),
and into the higher layers as an input signal (bottom-up feedforward
signaling). Rewards modulate the learning rate in those modules to memorize
meaningful sequences effectively. In mHPM, each module updates in a local and
distributed way compared to conventional end-to-end learning with
backpropagation of the single objective loss. This local structure enables the
heterarchical network of modules. The second type is an innate, special-purpose
module representing various organs of the brain's subcortical system. Modules
modeling organs such as the amygdala, hippocampus, and reward center are
pre-programmed to enable instinctive behaviors. The hippocampus plays the role
of the simulator. It is an autoregressive prediction model of the top-most
level signal with a loop structure of memory, while cortical columns are lower
layers that provide detailed information to the simulation. The simulation
becomes the basis for learning with few trials and the deliberate planning
required for operant conditioning.
