  This study presents a novel deep learning architecture for multi-class
classification and localization of abnormalities in medical imaging illustrated
through experiments on mammograms. The proposed network combines two learning
branches. One branch is for region classification with a newly added
normal-region class. Second branch is region detection branch for ranking
regions relative to one another. Our method enables detection of abnormalities
at full mammogram resolution for both weakly and semi-supervised settings. A
novel objective function allows for the incorporation of local annotations into
the model. We present the impact of our schemes on several performance measures
for classification and localization, to evaluate the cost effectiveness of the
lesion annotation effort. Our evaluation was primarily conducted over a large
multi-center mammography dataset of $\sim$3,000 mammograms with various
findings. The results for weakly supervised learning showed significant
improvement compared to previous approaches. We show that the time consuming
local annotations involved in supervised learning can be addressed by a weakly
supervised method that can leverage a subset of locally annotated data. Weakly
and semi-supervised methods coupled with detection can produce a cost effective
and explainable model to be adopted by radiologists in the field.
