  Accurate breast lesion risk estimation can significantly reduce unnecessary
biopsies and help doctors decide optimal treatment plans. Most existing
computer-aided systems rely solely on mammogram features to classify breast
lesions. While this approach is convenient, it does not fully exploit useful
information in clinical reports to achieve the optimal performance. Would
clinical features significantly improve breast lesion classification compared
to using mammograms alone? How to handle missing clinical information caused by
variation in medical practice? What is the best way to combine mammograms and
clinical features? There is a compelling need for a systematic study to address
these fundamental questions. This paper investigates several multimodal deep
networks based on feature concatenation, cross-attention, and co-attention to
combine mammograms and categorical clinical variables. We show that the
proposed architectures significantly increase the lesion classification
performance (average area under ROC curves from 0.89 to 0.94). We also evaluate
the model when clinical variables are missing.
