  Recent results on optimization and generalization properties of neural
networks showed that in a simple two-layer network, the alignment of the labels
to the eigenvectors of the corresponding Gram matrix determines the convergence
of the optimization during training. Such analyses also provide upper bounds on
the generalization error. We experimentally investigate the implications of
these results to deeper networks via embeddings. We regard the layers preceding
the final hidden layer as producing different representations of the input data
which are then fed to the two-layer model. We show that these representations
improve both optimization and generalization. In particular, we investigate
three kernel representations when fed to the final hidden layer: the Gaussian
kernel and its approximation by random Fourier features, kernels designed to
imitate representations produced by neural networks and finally an optimal
kernel designed to align the data with target labels. The approximated
representations induced by these kernels are fed to the neural network and the
optimization and generalization properties of the final model are evaluated and
compared.
