  Limiting failures of machine learning systems is of paramount importance for
safety-critical applications. In order to improve the robustness of machine
learning systems, Distributionally Robust Optimization (DRO) has been proposed
as a generalization of Empirical Risk Minimization (ERM). However, its use in
deep learning has been severely restricted due to the relative inefficiency of
the optimizers available for DRO in comparison to the wide-spread variants of
Stochastic Gradient Descent (SGD) optimizers for ERM. We propose SGD with
hardness weighted sampling, a principled and efficient optimization method for
DRO in machine learning that is particularly suited in the context of deep
learning. Similar to a hard example mining strategy in practice, the proposed
algorithm is straightforward to implement and computationally as efficient as
SGD-based optimizers used for deep learning, requiring minimal overhead
computation. In contrast to typical ad hoc hard mining approaches, we prove the
convergence of our DRO algorithm for over-parameterized deep learning networks
with ReLU activation and a finite number of layers and parameters. Our
experiments on fetal brain 3D MRI segmentation and brain tumor segmentation in
MRI demonstrate the feasibility and the usefulness of our approach. Using our
hardness weighted sampling for training a state-of-the-art deep learning
pipeline leads to improved robustness to anatomical variabilities in automatic
fetal brain 3D MRI segmentation using deep learning and to improved robustness
to the image protocol variations in brain tumor segmentation. Our code is
available at https://github.com/LucasFidon/HardnessWeightedSampler.
