  Neural networks are an indispensable model class for many complex learning
tasks. Despite the popularity and importance of neural networks and many
different established techniques from literature for stabilization and
robustification of the training, the classical concepts from robust statistics
have rarely been considered so far in the context of neural networks.
Therefore, we adapt the notion of the regression breakdown point to regression
neural networks and compute the breakdown point for different feed-forward
network configurations and contamination settings. In an extensive simulation
study, we compare the performance, measured by the out-of-sample loss, by a
proxy of the breakdown rate and by the training steps, of non-robust and robust
regression feed-forward neural networks in a plethora of different
configurations. The results indeed motivate to use robust loss functions for
neural network training.
