  In medical imaging, technical progress or changes in diagnostic procedures
lead to a continuous change in image appearance. Scanner manufacturer,
reconstruction kernel, dose, other protocol specific settings or administering
of contrast agents are examples that influence image content independent of the
scanned biology. Such domain and task shifts limit the applicability of machine
learning algorithms in the clinical routine by rendering models obsolete over
time. Here, we address the problem of data shifts in a continuous learning
scenario by adapting a model to unseen variations in the source domain while
counteracting catastrophic forgetting effects. Our method uses a dynamic memory
to facilitate rehearsal of a diverse training data subset to mitigate
forgetting. We evaluated our approach on routine clinical CT data obtained with
two different scanner protocols and synthetic classification tasks. Experiments
show that dynamic memory counters catastrophic forgetting in a setting with
multiple data shifts without the necessity for explicit knowledge about when
these shifts occur.
