  Congenital heart disease (CHD) is the most common type of birth defect, which
occurs 1 in every 110 births in the United States. CHD usually comes with
severe variations in heart structure and great artery connections that can be
classified into many types. Thus highly specialized domain knowledge and the
time-consuming human process is needed to analyze the associated medical
images. On the other hand, due to the complexity of CHD and the lack of
dataset, little has been explored on the automatic diagnosis (classification)
of CHDs. In this paper, we present ImageCHD, the first medical image dataset
for CHD classification. ImageCHD contains 110 3D Computed Tomography (CT)
images covering most types of CHD, which is of decent size Classification of
CHDs requires the identification of large structural changes without any local
tissue changes, with limited data. It is an example of a larger class of
problems that are quite difficult for current machine-learning-based vision
methods to solve. To demonstrate this, we further present a baseline framework
for the automatic classification of CHD, based on a state-of-the-art CHD
segmentation method. Experimental results show that the baseline framework can
only achieve a classification accuracy of 82.0\% under a selective prediction
scheme with 88.4\% coverage, leaving big room for further improvement. We hope
that ImageCHD can stimulate further research and lead to innovative and generic
solutions that would have an impact in multiple domains. Our dataset is
released to the public compared with existing medical imaging datasets.
