  We propose a simple but effective data-driven channel pruning algorithm,
which compresses deep neural networks in a differentiable way by exploiting the
characteristics of operations. The proposed approach makes a joint
consideration of batch normalization (BN) and rectified linear unit (ReLU) for
channel pruning; it estimates how likely the two successive operations
deactivate each feature map and prunes the channels with high probabilities. To
this end, we learn differentiable masks for individual channels and make soft
decisions throughout the optimization procedure, which facilitates to explore
larger search space and train more stable networks. The proposed framework
enables us to identify compressed models via a joint learning of model
parameters and channel pruning without an extra procedure of fine-tuning. We
perform extensive experiments and achieve outstanding performance in terms of
the accuracy of output networks given the same amount of resources when
compared with the state-of-the-art methods.
