  Regularized empirical risk minimization using kernels and their corresponding
reproducing kernel Hilbert spaces (RKHSs) plays an important role in machine
learning. However, the actually used kernel often depends on one or on a few
hyperparameters or the kernel is even data dependent in a much more complicated
manner. Examples are Gaussian RBF kernels, kernel learning, and hierarchical
Gaussian kernels which were recently proposed for deep learning. Therefore, the
actually used kernel is often computed by a grid search or in an iterative
manner and can often only be considered as an approximation to the "ideal" or
"optimal" kernel. The paper gives conditions under which classical kernel based
methods based on a convex Lipschitz loss function and on a bounded and smooth
kernel are stable, if the probability measure $P$, the regularization parameter
$\lambda$, and the kernel $k$ may slightly change in a simultaneous manner.
Similar results are also given for pairwise learning. Therefore, the topic of
this paper is somewhat more general than in classical robust statistics, where
usually only the influence of small perturbations of the probability measure
$P$ on the estimated function is considered.
