  We propose a novel deep neural network architecture for normalcy detection in
chest X-ray images. This architecture treats the problem as fine-grained binary
classification in which the normal cases are well-defined as a class while
leaving all other cases in the broad class of abnormal. It employs several
components that allow generalization and prevent overfitting across
demographics. The model is trained and validated on a large public dataset of
frontal chest X-ray images. It is then tested independently on images from a
clinical institution of differing patient demographics using a three
radiologist consensus for ground truth labeling. The model provides an area
under ROC curve of 0.96 when tested on 1271 images. We can automatically remove
nearly a third of disease-free chest X-ray screening images from the workflow,
without introducing any false negatives (100% sensitivity to disease) thus
raising the potential of expediting radiology workflows in hospitals in future.
