  We show that the stochasticity in training ResNets for image classification
on GPUs in TensorFlow is dominated by the non-determinism from GPUs, rather
than by the initialisation of the weights and biases of the network or by the
sequence of minibatches given. The standard deviation of test set accuracy is
0.02 with fixed seeds, compared to 0.027 with different seeds---nearly 74\% of
the standard deviation of a ResNet model is non-deterministic. For test set
loss the ratio of standard deviations is more than 80\%. These results call for
more robust evaluation strategies of deep learning models, as a significant
amount of the variation in results across runs can arise simply from GPU
randomness.
