  We analyze the dynamics of training deep ReLU networks and their implications
on generalization capability. Using a teacher-student setting, we discovered a
novel relationship between the gradient received by hidden student nodes and
the activations of teacher nodes for deep ReLU networks. With this relationship
and the assumption of small overlapping teacher node activations, we prove that
(1) student nodes whose weights are initialized to be close to teacher nodes
converge to them at a faster rate, and (2) in over-parameterized regimes and
2-layer case, while a small set of lucky nodes do converge to the teacher
nodes, the fan-out weights of other nodes converge to zero. This framework
provides insight into multiple puzzling phenomena in deep learning like
over-parameterization, implicit regularization, lottery tickets, etc. We verify
our assumption by showing that the majority of BatchNorm biases of pre-trained
VGG11/16 models are negative. Experiments on (1) random deep teacher networks
with Gaussian inputs, (2) teacher network pre-trained on CIFAR-10 and (3)
extensive ablation studies validate our multiple theoretical predictions.
