  Recent works show that reducing the number of layers in a convolutional
neural network can enhance efficiency while maintaining the performance of the
network. Existing depth compression methods remove redundant non-linear
activation functions and merge the consecutive convolution layers into a single
layer. However, these methods suffer from a critical drawback; the kernel size
of the merged layers becomes larger, significantly undermining the latency
reduction gained from reducing the depth of the network. We show that this
problem can be addressed by jointly pruning convolution layers and activation
functions. To this end, we propose LayerMerge, a novel depth compression method
that selects which activation layers and convolution layers to remove, to
achieve a desired inference speed-up while minimizing performance loss. Since
the corresponding selection problem involves an exponential search space, we
formulate a novel surrogate optimization problem and efficiently solve it via
dynamic programming. Empirical results demonstrate that our method consistently
outperforms existing depth compression and layer pruning methods on various
network architectures, both on image classification and generation tasks. We
release the code at https://github.com/snu-mllab/LayerMerge.
