  How does our motor system solve the problem of anticipatory control in spite
of a wide spectrum of response dynamics from different musculo-skeletal
systems, transport delays as well as response latencies throughout the central
nervous system? To a great extent, our highly-skilled motor responses are a
result of a reactive feedback system, originating in the brain-stem and spinal
cord, combined with a feed-forward anticipatory system, that is adaptively
fine-tuned by sensory experience and originates in the cerebellum. Based on
that interaction we design the counterfactual predictive control (CFPC)
architecture, an anticipatory adaptive motor control scheme in which a
feed-forward module, based on the cerebellum, steers an error feedback
controller with counterfactual error signals. Those are signals that trigger
reactions as actual errors would, but that do not code for any current or
forthcoming errors. In order to determine the optimal learning strategy, we
derive a novel learning rule for the feed-forward module that involves an
eligibility trace and operates at the synaptic level. In particular, our
eligibility trace provides a mechanism beyond co-incidence detection in that it
convolves a history of prior synaptic inputs with error signals. In the context
of cerebellar physiology, this solution implies that Purkinje cell synapses
should generate eligibility traces using a forward model of the system being
controlled. From an engineering perspective, CFPC provides a general-purpose
anticipatory control architecture equipped with a learning rule that exploits
the full dynamics of the closed-loop system.
