  Convolutional Neural Networks (CNNs) are typically constructed by stacking
multiple building blocks, each of which contains a normalization layer such as
batch normalization (BN) and a rectified linear function such as ReLU. However,
this work shows that the combination of normalization and rectified linear
function leads to inhibited channels, which have small magnitude and contribute
little to the learned feature representation, impeding the generalization
ability of CNNs. Unlike prior arts that simply removed the inhibited channels,
we propose to "wake them up" during training by designing a novel neural
building block, termed Channel Equilibrium (CE) block, which enables channels
at the same layer to contribute equally to the learned representation. We show
that CE is able to prevent inhibited channels both empirically and
theoretically. CE has several appealing benefits. (1) It can be integrated into
many advanced CNN architectures such as ResNet and MobileNet, outperforming
their original networks. (2) CE has an interesting connection with the Nash
Equilibrium, a well-known solution of a non-cooperative game. (3) Extensive
experiments show that CE achieves state-of-the-art performance on various
challenging benchmarks such as ImageNet and COCO.
