  Chest radiography is the most common medical image examination for screening
and diagnosis in hospitals. Automatic interpretation of chest X-rays at the
level of an entry-level radiologist can greatly benefit work prioritization and
assist in analyzing a larger population. Subsequently, several datasets and
deep learning-based solutions have been proposed to identify diseases based on
chest X-ray images. However, these methods are shown to be vulnerable to shift
in the source of data: a deep learning model performing well when tested on the
same dataset as training data, starts to perform poorly when it is tested on a
dataset from a different source. In this work, we address this challenge of
generalization to a new source by forcing the network to learn a
source-invariant representation. By employing an adversarial training strategy,
we show that a network can be forced to learn a source-invariant
representation. Through pneumonia-classification experiments on multi-source
chest X-ray datasets, we show that this algorithm helps in improving
classification accuracy on a new source of X-ray dataset.
