  Animals thrive in a constantly changing environment and leverage the temporal
structure to learn well-factorized causal representations. In contrast,
traditional neural networks suffer from forgetting in changing environments and
many methods have been proposed to limit forgetting with different trade-offs.
Inspired by the brain thalamocortical circuit, we introduce a simple algorithm
that uses optimization at inference time to generate internal representations
of the current task dynamically. The algorithm alternates between updating the
model weights and a latent task embedding, allowing the agent to parse the
stream of temporal experience into discrete events and organize learning about
them. On a continual learning benchmark, it achieves competitive end average
accuracy by mitigating forgetting, but importantly, by requiring the model to
adapt through latent updates, it organizes knowledge into flexible structures
with a cognitive interface to control them. Tasks later in the sequence can be
solved through knowledge transfer as they become reachable within the
well-factorized latent space. The algorithm meets many of the desiderata of an
ideal continually learning agent in open-ended environments, and its simplicity
suggests fundamental computations in circuits with abundant feedback control
loops such as the thalamocortical circuits in the brain.
