  In radiologists' routine work, one major task is to read a medical image,
e.g., a CT scan, find significant lesions, and describe them in the radiology
report. In this paper, we study the lesion description or annotation problem.
Given a lesion image, our aim is to predict a comprehensive set of relevant
labels, such as the lesion's body part, type, and attributes, which may assist
downstream fine-grained diagnosis. To address this task, we first design a deep
learning module to extract relevant semantic labels from the radiology reports
associated with the lesion images. With the images and text-mined labels, we
propose a lesion annotation network (LesaNet) based on a multilabel
convolutional neural network (CNN) to learn all labels holistically.
Hierarchical relations and mutually exclusive relations between the labels are
leveraged to improve the label prediction accuracy. The relations are utilized
in a label expansion strategy and a relational hard example mining algorithm.
We also attach a simple score propagation layer on LesaNet to enhance recall
and explore implicit relation between labels. Multilabel metric learning is
combined with classification to enable interpretable prediction. We evaluated
LesaNet on the public DeepLesion dataset, which contains over 32K diverse
lesion images. Experiments show that LesaNet can precisely annotate the lesions
using an ontology of 171 fine-grained labels with an average AUC of 0.9344.
