  We propose a novel approach to achieving invariance for deep neural networks
in the form of inducing amnesia to unwanted factors of data through a new
adversarial forgetting mechanism. We show that the forgetting mechanism serves
as an information-bottleneck, which is manipulated by the adversarial training
to learn invariance to unwanted factors. Empirical results show that the
proposed framework achieves state-of-the-art performance at learning invariance
in both nuisance and bias settings on a diverse collection of datasets and
tasks.
