  We consider kernels of the form $(x,x') \mapsto \phi(\|x-x'\|^2_\Sigma)$
parametrized by $\Sigma$. For such kernels, we study a variant of the kernel
ridge regression problem which simultaneously optimizes the prediction function
and the parameter $\Sigma$ of the reproducing kernel Hilbert space. The
eigenspace of the $\Sigma$ learned from this kernel ridge regression problem
can inform us which directions in covariate space are important for prediction.
  Assuming that the covariates have nonzero explanatory power for the response
only through a low dimensional subspace (central mean subspace), we find that
the global minimizer of the finite sample kernel learning objective is also low
rank with high probability. More precisely, the rank of the minimizing $\Sigma$
is with high probability bounded by the dimension of the central mean subspace.
This phenomenon is interesting because the low rankness property is achieved
without using any explicit regularization of $\Sigma$, e.g., nuclear norm
penalization.
  Our theory makes correspondence between the observed phenomenon and the
notion of low rank set identifiability from the optimization literature. The
low rankness property of the finite sample solutions exists because the
population kernel learning objective grows "sharply" when moving away from its
minimizers in any direction perpendicular to the central mean subspace.
