  Recent extensive numerical experiments in high scale machine learning have
allowed to uncover a quite counterintuitive phase transition, as a function of
the ratio between the sample size and the number of parameters in the model. As
the number of parameters $p$ approaches the sample size $n$, the generalisation
error increases, but surprisingly, it starts decreasing again past the
threshold $p=n$. This phenomenon, brought to the theoretical community
attention in \cite{belkin2019reconciling}, has been thoroughly investigated
lately, more specifically for simpler models than deep neural networks, such as
the linear model when the parameter is taken to be the minimum norm solution to
the least-squares problem, firstly in the asymptotic regime when $p$ and $n$
tend to infinity, see e.g. \cite{hastie2019surprises}, and recently in the
finite dimensional regime and more specifically for linear models
\cite{bartlett2020benign}, \cite{tsigler2020benign},
\cite{lecue2022geometrical}. In the present paper, we propose a finite sample
analysis of non-linear models of \textit{ridge} type, where we investigate the
\textit{overparametrised regime} of the double descent phenomenon for both the
\textit{estimation problem} and the \textit{prediction} problem. Our results
provide a precise analysis of the distance of the best estimator from the true
parameter as well as a generalisation bound which complements recent works of
\cite{bartlett2020benign} and \cite{chinot2020benign}. Our analysis is based on
tools closely related to the continuous Newton method
\cite{neuberger2007continuous} and a refined quantitative analysis of the
performance in prediction of the minimum $\ell_2$-norm solution.
