  Recent work in incremental learning has introduced diverse approaches to
tackle catastrophic forgetting from data augmentation to optimized training
regimes. However, most of them focus on very few training steps. We propose a
method for robust incremental learning over dozens of fine-tuning steps using
data from a variety of languages. We show that a combination of
data-augmentation and an optimized training regime allows us to continue
improving the model even for as many as fifty training steps. Crucially, our
augmentation strategy does not require retaining access to previous training
data and is suitable in scenarios with privacy constraints.
