  We propose a data-driven approach to quantify the uncertainty of models
constructed by kernel methods. Our approach minimizes the needed distributional
assumptions, hence, instead of working with, for example, Gaussian processes or
exponential families, it only requires knowledge about some mild regularity of
the measurement noise, such as it is being symmetric or exchangeable. We show,
by building on recent results from finite-sample system identification, that by
perturbing the residuals in the gradient of the objective function, information
can be extracted about the amount of uncertainty our model has. Particularly,
we provide an algorithm to build exact, non-asymptotically guaranteed,
distribution-free confidence regions for ideal, noise-free representations of
the function we try to estimate. For the typical convex quadratic problems and
symmetric noises, the regions are star convex centered around a given nominal
estimate, and have efficient ellipsoidal outer approximations. Finally, we
illustrate the ideas on typical kernel methods, such as LS-SVC, KRR,
$\varepsilon$-SVR and kernelized LASSO.
