  We develop an end-to-end training algorithm for whole-image breast cancer
diagnosis based on mammograms. It requires lesion annotations only at the first
stage of training. After that, a whole image classifier can be trained using
only image level labels. This greatly reduced the reliance on lesion
annotations. Our approach is implemented using an all convolutional design that
is simple yet provides superior performance in comparison with the previous
methods. On DDSM, our best single-model achieves a per-image AUC score of 0.88
and three-model averaging increases the score to 0.91. On INbreast, our best
single-model achieves a per-image AUC score of 0.96. Using DDSM as benchmark,
our models compare favorably with the current state-of-the-art. We also
demonstrate that a whole image model trained on DDSM can be easily transferred
to INbreast without using its lesion annotations and using only a small amount
of training data. Code availability: https://github.com/lishen/end2end-all-conv
