  The celebrated Nadaraya-Watson kernel estimator is among the most studied
method for nonparametric regression. A classical result is that its rate of
convergence depends on the number of covariates and deteriorates quickly as the
dimension grows, which underscores the "curse of dimensionality" and has
limited its use in high dimensional settings. In this article, we show that
when the true regression function is single or multi-index, the effects of the
curse of dimensionality may be mitigated for the Nadaraya-Watson kernel
estimator. Specifically, we prove that with $K$-fold cross-validation, the
Nadaraya-Watson kernel estimator indexed by a positive semidefinite bandwidth
matrix has an oracle property that its rate of convergence depends on the
number of indices of the regression function rather than the number of
covariates. Intuitively, this oracle property is a consequence of allowing the
bandwidths to diverge to infinity as opposed to restricting them all to
converge to zero at certain rates as done in previous theoretical studies. Our
result provides a theoretical perspective for the use of kernel estimation in
high dimensional nonparametric regression and other applications such as metric
learning when a low rank structure is anticipated. Numerical illustrations are
given through simulations and real data examples.
