  The kernel trick concept, formulated as an inner product in a feature space,
facilitates powerful extensions to many well-known algorithms. While the kernel
matrix involves inner products in the feature space, the sample covariance
matrix of the data requires outer products. Therefore, their spectral
properties are tightly connected. This allows us to examine the kernel matrix
through the sample covariance matrix in the feature space and vice versa. The
use of kernels often involves a large number of features, compared to the
number of observations. In this scenario, the sample covariance matrix is not
well-conditioned nor is it necessarily invertible, mandating a solution to the
problem of estimating high-dimensional covariance matrices under small sample
size conditions. We tackle this problem through the use of a shrinkage
estimator that offers a compromise between the sample covariance matrix and a
well-conditioned matrix (also known as the "target") with the aim of minimizing
the mean-squared error (MSE). We propose a distribution-free kernel matrix
regularization approach that is tuned directly from the kernel matrix, avoiding
the need to address the feature space explicitly. Numerical simulations
demonstrate that the proposed regularization is effective in classification
tasks.
