  Inner product-based convolution has been a central component of convolutional
neural networks (CNNs) and the key to learning visual representations. Inspired
by the observation that CNN-learned features are naturally decoupled with the
norm of features corresponding to the intra-class variation and the angle
corresponding to the semantic difference, we propose a generic decoupled
learning framework which models the intra-class variation and semantic
difference independently. Specifically, we first reparametrize the inner
product to a decoupled form and then generalize it to the decoupled convolution
operator which serves as the building block of our decoupled networks. We
present several effective instances of the decoupled convolution operator. Each
decoupled operator is well motivated and has an intuitive geometric
interpretation. Based on these decoupled operators, we further propose to
directly learn the operator from data. Extensive experiments show that such
decoupled reparameterization renders significant performance gain with easier
convergence and stronger robustness.
