  Depth is one of the keys that make neural networks succeed in the task of
large-scale image recognition. The state-of-the-art network architectures
usually increase the depths by cascading convolutional layers or building
blocks. In this paper, we present an alternative method to increase the depth.
Our method is by introducing computation orderings to the channels within
convolutional layers or blocks, based on which we gradually compute the outputs
in a channel-wise manner. The added orderings not only increase the depths and
the learning capacities of the networks without any additional computation
costs, but also eliminate the overlap singularities so that the networks are
able to converge faster and perform better. Experiments show that the networks
based on our method achieve the state-of-the-art performances on CIFAR and
ImageNet datasets.
