  In radiologists' routine work, one major task is to read a medical image,
e.g., a CT scan, find significant lesions, and write sentences in the radiology
report to describe them. In this paper, we study the lesion description or
annotation problem as an important step of computer-aided diagnosis (CAD).
Given a lesion image, our aim is to predict multiple relevant labels, such as
the lesion's body part, type, and attributes. To address this problem, we
define a set of 145 labels based on RadLex to describe a large variety of
lesions in the DeepLesion dataset. We directly mine training labels from the
lesion's corresponding sentence in the radiology report, which requires minimal
manual effort and is easily generalizable to large data and label sets. A
multi-label convolutional neural network is then proposed for images with
multi-scale structure and a noise-robust loss. Quantitative and qualitative
experiments demonstrate the effectiveness of the framework. The average area
under ROC curve on 1,872 test lesions is 0.9083.
