  The notion of developing statistical methods in machine learning which are
robust to adversarial perturbations in the underlying data has been the subject
of increasing interest in recent years. A common feature of this work is that
the adversarial robustification often corresponds exactly to regularization
methods which appear as a loss function plus a penalty. In this paper we deepen
and extend the understanding of the connection between robustification and
regularization (as achieved by penalization) in regression problems.
Specifically, (a) in the context of linear regression, we characterize
precisely under which conditions on the model of uncertainty used and on the
loss function penalties robustification and regularization are equivalent, and
(b) we extend the characterization of robustification and regularization to
matrix regression problems (matrix completion and Principal Component
Analysis).
