  This paper studies the novel concept of weight correlation in deep neural
networks and discusses its impact on the networks' generalisation ability. For
fully-connected layers, the weight correlation is defined as the average cosine
similarity between weight vectors of neurons, and for convolutional layers, the
weight correlation is defined as the cosine similarity between filter matrices.
Theoretically, we show that, weight correlation can, and should, be
incorporated into the PAC Bayesian framework for the generalisation of neural
networks, and the resulting generalisation bound is monotonic with respect to
the weight correlation. We formulate a new complexity measure, which lifts the
PAC Bayes measure with weight correlation, and experimentally confirm that it
is able to rank the generalisation errors of a set of networks more precisely
than existing measures. More importantly, we develop a new regulariser for
training, and provide extensive experiments that show that the generalisation
error can be greatly reduced with our novel approach.
