  Regularization is key for deep learning since it allows training more complex
models while keeping lower levels of overfitting. However, the most prevalent
regularizations do not leverage all the capacity of the models since they rely
on reducing the effective number of parameters. Feature decorrelation is an
alternative for using the full capacity of the models but the overfitting
reduction margins are too narrow given the overhead it introduces. In this
paper, we show that regularizing negatively correlated features is an obstacle
for effective decorrelation and present OrthoReg, a novel regularization
technique that locally enforces feature orthogonality. As a result, imposing
locality constraints in feature decorrelation removes interferences between
negatively correlated feature weights, allowing the regularizer to reach higher
decorrelation bounds, and reducing the overfitting more effectively. In
particular, we show that the models regularized with OrthoReg have higher
accuracy bounds even when batch normalization and dropout are present.
Moreover, since our regularization is directly performed on the weights, it is
especially suitable for fully convolutional neural networks, where the weight
space is constant compared to the feature map space. As a result, we are able
to reduce the overfitting of state-of-the-art CNNs on CIFAR-10, CIFAR-100, and
SVHN.
