  We propose a novel family of connectionist models based on kernel machines
and consider the problem of learning layer-by-layer a compositional hypothesis
class, i.e., a feedforward, multilayer architecture, in a supervised setting.
In terms of the models, we present a principled method to "kernelize" (partly
or completely) any neural network (NN). With this method, we obtain a
counterpart of any given NN that is powered by kernel machines instead of
neurons. In terms of learning, when learning a feedforward deep architecture in
a supervised setting, one needs to train all the components simultaneously
using backpropagation (BP) since there are no explicit targets for the hidden
layers (Rumelhart86). We consider without loss of generality the two-layer case
and present a general framework that explicitly characterizes a target for the
hidden layer that is optimal for minimizing the objective function of the
network. This characterization then makes possible a purely greedy training
scheme that learns one layer at a time, starting from the input layer. We
provide realizations of the abstract framework under certain architectures and
objective functions. Based on these realizations, we present a layer-wise
training algorithm for an l-layer feedforward network for classification, where
l>=2 can be arbitrary. This algorithm can be given an intuitive geometric
interpretation that makes the learning dynamics transparent. Empirical results
are provided to complement our theory. We show that the kernelized networks,
trained layer-wise, compare favorably with classical kernel machines as well as
other connectionist models trained by BP. We also visualize the inner workings
of the greedy kernelized models to validate our claim on the transparency of
the layer-wise algorithm.
