  Due to the limited availability of medical data, deep learning approaches for
medical image analysis tend to generalise poorly to unseen data. Augmenting
data during training with random transformations has been shown to help and
became a ubiquitous technique for training neural networks. Here, we propose a
novel adversarial counterfactual augmentation scheme that aims at finding the
most \textit{effective} synthesised images to improve downstream tasks, given a
pre-trained generative model. Specifically, we construct an adversarial game
where we update the input \textit{conditional factor} of the generator and the
downstream \textit{classifier} with gradient backpropagation alternatively and
iteratively. This can be viewed as finding the `\textit{weakness}' of the
classifier and purposely forcing it to \textit{overcome} its weakness via the
generative model. To demonstrate the effectiveness of the proposed approach, we
validate the method with the classification of Alzheimer's Disease (AD) as a
downstream task. The pre-trained generative model synthesises brain images
using age as conditional factor. Extensive experiments and ablation studies
have been performed to show that the proposed approach improves classification
performance and has potential to alleviate spurious correlations and
catastrophic forgetting. Code will be released upon acceptance.
