  Learning algorithms based on backpropagation have enabled transformative
technological advances but alternatives based on local energy-based rules offer
benefits in terms of biological plausibility and decentralized training. A
broad class of such local learning rules involve \textit{contrasting} a clamped
configuration with the free, spontaneous behavior of the system. However,
comparisons of clamped and free configurations require explicit memory or
switching between Hebbian and anti-Hebbian modes. Here, we show how a simple
form of implicit non-equilibrium memory in the update dynamics of each
``synapse'' of a network naturally allows for contrastive learning. During
training, free and clamped behaviors are shown in sequence over time using a
sawtooth-like temporal protocol that breaks the symmetry between those two
behaviors when combined with non-equilibrium update dynamics at each synapse.
We show that the needed dynamics is implicit in integral feedback control,
broadening the range of physical and biological systems naturally capable of
contrastive learning. Finally, we show that non-equilibrium dissipation
improves learning quality and determine the Landauer energy cost of contrastive
learning through physical dynamics.
