  Multilingual speech recognition with neural networks is often implemented
with batch-learning, when all of the languages are available before training.
An ability to add new languages after the prior training sessions can be
economically beneficial, but the main challenge is catastrophic forgetting. In
this work, we combine the qualities of weight factorization and elastic weight
consolidation in order to counter catastrophic forgetting and facilitate
learning new languages quickly. Such combination allowed us to eliminate
catastrophic forgetting while still achieving performance for the new languages
comparable with having all languages at once, in experiments of learning from
an initial 10 languages to achieve 26 languages without catastrophic forgetting
and a reasonable performance compared to training all languages from scratch.
