  Temporal difference (TD) learning is often used to update the estimate of the
value function which is used by RL agents to extract useful policies. In this
paper, we focus on value function estimation in continual reinforcement
learning. We propose to decompose the value function into two components which
update at different timescales: a permanent value function, which holds general
knowledge that persists over time, and a transient value function, which allows
quick adaptation to new situations. We establish theoretical results showing
that our approach is well suited for continual learning and draw connections to
the complementary learning systems (CLS) theory from neuroscience. Empirically,
this approach improves performance significantly on both prediction and control
problems.
