  Popular machine learning estimators involve regularization parameters that
can be challenging to tune, and standard strategies rely on grid search for
this task. In this paper, we revisit the techniques of approximating the
regularization path up to predefined tolerance $\epsilon$ in a unified
framework and show that its complexity is $O(1/\sqrt[d]{\epsilon})$ for
uniformly convex loss of order $d \geq 2$ and $O(1/\sqrt{\epsilon})$ for
Generalized Self-Concordant functions. This framework encompasses least-squares
but also logistic regression, a case that as far as we know was not handled as
precisely in previous works. We leverage our technique to provide refined
bounds on the validation error as well as a practical algorithm for
hyperparameter tuning. The latter has global convergence guarantee when
targeting a prescribed accuracy on the validation set. Last but not least, our
approach helps relieving the practitioner from the (often neglected) task of
selecting a stopping criterion when optimizing over the training set: our
method automatically calibrates this criterion based on the targeted accuracy
on the validation set.
