  Although deep learning can provide promising results in medical image
analysis, the lack of very large annotated datasets confines its full
potential. Furthermore, limited positive samples also create unbalanced
datasets which limit the true positive rates of trained models. As unbalanced
datasets are mostly unavoidable, it is greatly beneficial if we can extract
useful knowledge from negative samples to improve classification accuracy on
limited positive samples. To this end, we propose a new strategy for building
medical image analysis pipelines that target disease detection. We train a
discriminative segmentation model only on normal images to provide a source of
knowledge to be transferred to a disease detection classifier. We show that
using the feature maps of a trained segmentation network, deviations from
normal anatomy can be learned by a two-class classification network on an
extremely unbalanced training dataset with as little as one positive for 17
negative samples. We demonstrate that even though the segmentation network is
only trained on normal cardiac computed tomography images, the resulting
feature maps can be used to detect pericardial effusion and cardiac septal
defects with two-class convolutional classification networks.
