  It has been observed that certain loss functions can render deep-learning
pipelines robust against flaws in the data. In this paper, we support these
empirical findings with statistical theory. We especially show that
empirical-risk minimization with unbounded, Lipschitz-continuous loss
functions, such as the least-absolute deviation loss, Huber loss, Cauchy loss,
and Tukey's biweight loss, can provide efficient prediction under minimal
assumptions on the data. More generally speaking, our paper provides
theoretical evidence for the benefits of robust loss functions in deep
learning.
