  Iterative regularization is a classic idea in regularization theory, that has
recently become popular in machine learning. On the one hand, it allows to
design efficient algorithms controlling at the same time numerical and
statistical accuracy. On the other hand it allows to shed light on the learning
curves observed while training neural networks. In this paper, we focus on
iterative regularization in the context of classification. After contrasting
this setting with that of regression and inverse problems, we develop an
iterative regularization approach based on the use of the hinge loss function.
More precisely we consider a diagonal approach for a family of algorithms for
which we prove convergence as well as rates of convergence. Our approach
compares favorably with other alternatives, as confirmed also in numerical
simulations.
