  Synthetic corruptions gathered into a benchmark are frequently used to
measure neural network robustness to distribution shifts. However, robustness
to synthetic corruption benchmarks is not always predictive of robustness to
distribution shifts encountered in real-world applications. In this paper, we
propose a methodology to build synthetic corruption benchmarks that make
robustness estimations more correlated with robustness to real-world
distribution shifts. Using the overlapping criterion, we split synthetic
corruptions into categories that help to better understand neural network
robustness. Based on these categories, we identify three relevant parameters to
take into account when constructing a corruption benchmark that are the (1)
number of represented categories, (2) their relative balance in terms of size
and, (3) the size of the considered benchmark. In doing so, we build new
synthetic corruption selections that are more predictive of robustness to
natural corruptions than existing synthetic corruption benchmarks.
