  Extracting, harvesting and building large-scale annotated radiological image
datasets is a greatly important yet challenging problem. It is also the
bottleneck to designing more effective data-hungry computing paradigms (e.g.,
deep learning) for medical image analysis. Yet, vast amounts of clinical
annotations (usually associated with disease image findings and marked using
arrows, lines, lesion diameters, segmentation, etc.) have been collected over
several decades and stored in hospitals' Picture Archiving and Communication
Systems. In this paper, we mine and harvest one major type of clinical
annotation data - lesion diameters annotated on bookmarked images - to learn an
effective multi-class lesion detector via unsupervised and supervised deep
Convolutional Neural Networks (CNN). Our dataset is composed of 33,688
bookmarked radiology images from 10,825 studies of 4,477 unique patients. For
every bookmarked image, a bounding box is created to cover the target lesion
based on its measured diameters. We categorize the collection of lesions using
an unsupervised deep mining scheme to generate clustered pseudo lesion labels.
Next, we adopt a regional-CNN method to detect lesions of multiple categories,
regardless of missing annotations (normally only one lesion is annotated,
despite the presence of multiple co-existing findings). Our integrated mining,
categorization and detection framework is validated with promising empirical
results, as a scalable, universal or multi-purpose CAD paradigm built upon
abundant retrospective medical data. Furthermore, we demonstrate that detection
accuracy can be significantly improved by incorporating pseudo lesion labels
(e.g., Liver lesion/tumor, Lung nodule/tumor, Abdomen lesions, Chest lymph node
and others). This dataset will be made publicly available (under the open
science initiative).
