  Training a robust system, e.g.,Speech to Text (STT), requires large datasets.
Variability present in the dataset such as unwanted nuisances and biases are
the reason for the need of large datasets to learn general representations. In
this work, we propose a novel approach to induce invariance using adversarial
forgetting (AF). Our initial experiments on learning invariant features such as
accent on the STT task achieve better generalizations in terms of word error
rate (WER) compared to the traditional models. We observe an absolute
improvement of 2.2% and 1.3% on out-of-distribution and in-distribution test
sets, respectively.
