  We investigate the high-dimensional properties of robust regression
estimators in the presence of heavy-tailed contamination of both the covariates
and response functions. In particular, we provide a sharp asymptotic
characterisation of M-estimators trained on a family of elliptical covariate
and noise data distributions including cases where second and higher moments do
not exist. We show that, despite being consistent, the Huber loss with
optimally tuned location parameter $\delta$ is suboptimal in the
high-dimensional regime in the presence of heavy-tailed noise, highlighting the
necessity of further regularisation to achieve optimal performance. This result
also uncovers the existence of a transition in $\delta$ as a function of the
sample complexity and contamination. Moreover, we derive the decay rates for
the excess risk of ridge regression. We show that, while it is both optimal and
universal for covariate distributions with finite second moment, its decay rate
can be considerably faster when the covariates' second moment does not exist.
Finally, we show that our formulas readily generalise to a richer family of
models and data distributions, such as generalised linear estimation with
arbitrary convex regularisation trained on mixture models.
