  A fundamental question in deep learning concerns the role played by
individual layers in a deep neural network (DNN) and the transferable
properties of the data representations which they learn. To the extent that
layers have clear roles, one should be able to optimize them separately using
layer-wise loss functions. Such loss functions would describe what is the set
of good data representations at each depth of the network and provide a target
for layer-wise greedy optimization (LEGO). Here we derive a novel
correspondence between Gaussian Processes and SGD trained deep neural networks.
Leveraging this correspondence, we derive the Deep Gaussian Layer-wise loss
functions (DGLs) which, we believe, are the first supervised layer-wise loss
functions which are both explicit and competitive in terms of accuracy. Being
highly structured and symmetric, the DGLs provide a promising analytic route to
understanding the internal representations generated by DNNs.
