  Deep neural models have shown remarkable performance in image recognition
tasks, whenever large datasets of labeled images are available. The largest
datasets in radiology are available for screening mammography. Recent reports,
including in high impact journals, document performance of deep models at or
above that of trained radiologists. What is not yet known is whether
performance of these trained models is robust and replicates across datasets.
Here we evaluate performance of five published state-of-the-art models on four
publicly available mammography datasets. The limited size of public datasets
precludes retraining the model and so we are limited to evaluate those models
that have been made available with pre-trained parameters. Where test data was
available, we replicated published results. However, the trained models
performed poorly on out-of-sample data, except when based on all four standard
views of a mammographic exam. We conclude that future progress will depend on a
concerted effort to make more diverse and larger mammography datasets publicly
available. Meanwhile, results that are not accompanied by a release of trained
models for independent validation should be judged cautiously.
