  M-estimators are ubiquitous in machine learning and statistical learning
theory. They are used both for defining prediction strategies and for
evaluating their precision. In this paper, we propose the first non-asymptotic
"any-time" deviation bounds for general M-estimators, where "any-time" means
that the bound holds with a prescribed probability for every sample size. These
bounds are nonasymptotic versions of the law of iterated logarithm. They are
established under general assumptions such as Lipschitz continuity of the loss
function and (local) curvature of the population risk. These conditions are
satisfied for most examples used in machine learning, including those ensuring
robustness to outliers and to heavy tailed distributions. As an example of
application, we consider the problem of best arm identification in a parametric
stochastic multi-arm bandit setting. We show that the established bound can be
converted into a new algorithm, with provably optimal theoretical guarantees.
Numerical experiments illustrating the validity of the algorithm are reported.
