  We propose a greedy strategy to spectrally train a deep network for
multi-class classification. Each layer is defined as a composition of linear
weights with the feature map of a Gaussian kernel acting as the activation
function. At each layer, the linear weights are learned by maximizing the
dependence between the layer output and the labels using the Hilbert Schmidt
Independence Criterion (HSIC). By constraining the solution space on the
Stiefel Manifold, we demonstrate how our network construct (Kernel Dependence
Network or KNet) can be solved spectrally while leveraging the eigenvalues to
automatically find the width and the depth of the network. We theoretically
guarantee the existence of a solution for the global optimum while providing
insight into our network's ability to generalize.
