  Replication studies are essential for validation of new methods, and are
crucial to maintain the high standards of scientific publications, and to use
the results in practice. We have attempted to replicate the main method in
'Development and validation of a deep learning algorithm for detection of
diabetic retinopathy in retinal fundus photographs' published in JAMA 2016;
316(22). We re-implemented the method since the source code is not available,
and we used publicly available data sets. The original study used non-public
fundus images from EyePACS and three hospitals in India for training. We used a
different EyePACS data set from Kaggle. The original study used the benchmark
data set Messidor-2 to evaluate the algorithm's performance. We used the same
data set. In the original study, ophthalmologists re-graded all images for
diabetic retinopathy, macular edema, and image gradability. There was one
diabetic retinopathy grade per image for our data sets, and we assessed image
gradability ourselves. Hyper-parameter settings were not described in the
original study. But some of these were later published. We were not able to
replicate the original study. Our algorithm's area under the receiver operating
curve (AUC) of 0.94 on the Kaggle EyePACS test set and 0.80 on Messidor-2 did
not come close to the reported AUC of 0.99 in the original study. This may be
caused by the use of a single grade per image, different data, or different not
described hyper-parameter settings. This study shows the challenges of
replicating deep learning, and the need for more replication studies to
validate deep learning methods, especially for medical image analysis.
  Our source code and instructions are available at:
https://github.com/mikevoets/jama16-retina-replication
