  The astounding success made by artificial intelligence (AI) in healthcare and
other fields proves that AI can achieve human-like performance. However,
success always comes with challenges. Deep learning algorithms are
data-dependent and require large datasets for training. The lack of data in the
medical imaging field creates a bottleneck for the application of deep learning
to medical image analysis. Medical image acquisition, annotation, and analysis
are costly, and their usage is constrained by ethical restrictions. They also
require many resources, such as human expertise and funding. That makes it
difficult for non-medical researchers to have access to useful and large
medical data. Thus, as comprehensive as possible, this paper provides a
collection of medical image datasets with their associated challenges for deep
learning research. We have collected information of around three hundred
datasets and challenges mainly reported between 2013 and 2020 and categorized
them into four categories: head & neck, chest & abdomen, pathology & blood, and
``others''. Our paper has three purposes: 1) to provide a most up to date and
complete list that can be used as a universal reference to easily find the
datasets for clinical image analysis, 2) to guide researchers on the
methodology to test and evaluate their methods' performance and robustness on
relevant datasets, 3) to provide a ``route'' to relevant algorithms for the
relevant medical topics, and challenge leaderboards.
