  We introduce the MNIST-C dataset, a comprehensive suite of 15 corruptions
applied to the MNIST test set, for benchmarking out-of-distribution robustness
in computer vision. Through several experiments and visualizations we
demonstrate that our corruptions significantly degrade performance of
state-of-the-art computer vision models while preserving the semantic content
of the test images. In contrast to the popular notion of adversarial
robustness, our model-agnostic corruptions do not seek worst-case performance
but are instead designed to be broad and diverse, capturing multiple failure
modes of modern models. In fact, we find that several previously published
adversarial defenses significantly degrade robustness as measured by MNIST-C.
We hope that our benchmark serves as a useful tool for future work in designing
systems that are able to learn robust feature representations that capture the
underlying semantics of the input.
