  We propose a collection of three shift-based primitives for building
efficient compact CNN-based networks. These three primitives (channel shift,
address shift, shortcut shift) can reduce the inference time on GPU while
maintains the prediction accuracy. These shift-based primitives only moves the
pointer but avoids memory copy, thus very fast. For example, the channel shift
operation is 12.7x faster compared to channel shuffle in ShuffleNet but
achieves the same accuracy. The address shift and channel shift can be merged
into the point-wise group convolution and invokes only a single kernel call,
taking little time to perform spatial convolution and channel shift. Shortcut
shift requires no time to realize residual connection through allocating space
in advance. We blend these shift-based primitives with point-wise group
convolution and built two inference-efficient CNN architectures named
AddressNet and Enhanced AddressNet. Experiments on CIFAR100 and ImageNet
datasets show that our models are faster and achieve comparable or better
accuracy.
