  This paper studies convergence of empirical risks in reproducing kernel
Hilbert spaces (RKHS). A conventional assumption in the existing research is
that empirical training data do not contain any noise but this may not be
satisfied in some practical circumstances. Consequently the existing
convergence results do not provide a guarantee as to whether empirical risks
based on empirical data are reliable or not when the data contain some noise.
In this paper, we fill out the gap in a few steps. First, we derive moderate
sufficient conditions under which the expected risk changes stably
(continuously) against small perturbation of the probability distribution of
the underlying random variables and demonstrate how the cost function and
kernel affect the stability. Second, we examine the difference between laws of
the statistical estimators of the expected optimal loss based on pure data and
contaminated data using Prokhorov metric and Kantorovich metric and derive some
qualitative and quantitative statistical robustness results. Third, we identify
appropriate metrics under which the statistical estimators are uniformly
asymptotically consistent. These results provide theoretical grounding for
analysing asymptotic convergence and examining reliability of the statistical
estimators in a number of well-known machine learning models.
