  Continual Learning addresses the challenge of learning a number of different
tasks sequentially. The goal of maintaining knowledge of earlier tasks without
re-accessing them starkly conflicts with standard SGD training for artificial
neural networks. An influential method to tackle this problem without storing
old data are so-called regularisation approaches. They measure the importance
of each parameter for solving a given task and subsequently protect important
parameters from large changes. In the literature, three ways to measure
parameter importance have been put forward and they have inspired a large body
of follow-up work. Here, we present strong theoretical and empirical evidence
that these three methods, Elastic Weight Consolidation (EWC), Synaptic
Intelligence (SI) and Memory Aware Synapses (MAS), are surprisingly similar and
are all linked to the same theoretical quantity. Concretely, we show that,
despite stemming from very different motivations, both SI and MAS approximate
the square root of the Fisher Information, with the Fisher being the
theoretically justified basis of EWC. Moreover, we show that for SI the
relation to the Fisher -- and in fact its performance -- is due to a previously
unknown bias. On top of uncovering unknown similarities and unifying
regularisation approaches, we also demonstrate that our insights enable
practical performance improvements for large batch training.
