  Fine-tuning an Automatic Speech Recognition (ASR) model to new domains
results in degradation on original domains, referred to as Catastrophic
Forgetting (CF). Continual Learning (CL) attempts to train ASR models without
suffering from CF. While in ASR, offline CL is usually considered, online CL is
a more realistic but also more challenging scenario where the model, unlike in
offline CL, does not know when a task boundary occurs. Rehearsal-based methods,
which store previously seen utterances in a memory, are often considered for
online CL, in ASR and other research domains. However, recent research has
shown that weight averaging is an effective method for offline CL in ASR. Based
on this result, we propose, in this paper, a rehearsal-free method applicable
for online CL. Our method outperforms all baselines, including rehearsal-based
methods, in two experiments. Our method is a next step towards general CL for
ASR, which should enable CL in all scenarios with few if any constraints.
