  In this paper, we are concerned with the generalization performance of
non-parametric estimation for pairwise learning. Most of the existing work
requires the hypothesis space to be convex or a VC-class, and the loss to be
convex. However, these restrictive assumptions limit the applicability of the
results in studying many popular methods, especially kernel methods and neural
networks. We significantly relax these restrictive assumptions and establish a
sharp oracle inequality of the empirical minimizer with a general hypothesis
space for the Lipschitz continuous pairwise losses. Our results can be used to
handle a wide range of pairwise learning problems including ranking, AUC
maximization, pairwise regression, and metric and similarity learning. As an
application, we apply our general results to study pairwise least squares
regression and derive an excess generalization bound that matches the minimax
lower bound for pointwise least squares regression up to a logrithmic term. The
key novelty here is to construct a structured deep ReLU neural network as an
approximation of the true predictor and design the targeted hypothesis space
consisting of the structured networks with controllable complexity. This
successful application demonstrates that the obtained general results indeed
help us to explore the generalization performance on a variety of problems that
cannot be handled by existing approaches.
