  Compared with cheap addition operation, multiplication operation is of much
higher computation complexity. The widely-used convolutions in deep neural
networks are exactly cross-correlation to measure the similarity between input
feature and convolution filters, which involves massive multiplications between
float values. In this paper, we present adder networks (AdderNets) to trade
these massive multiplications in deep neural networks, especially convolutional
neural networks (CNNs), for much cheaper additions to reduce computation costs.
In AdderNets, we take the $\ell_1$-norm distance between filters and input
feature as the output response. The influence of this new similarity measure on
the optimization of neural network have been thoroughly analyzed. To achieve a
better performance, we develop a special back-propagation approach for
AdderNets by investigating the full-precision gradient. We then propose an
adaptive learning rate strategy to enhance the training procedure of AdderNets
according to the magnitude of each neuron's gradient. As a result, the proposed
AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50
on the ImageNet dataset without any multiplication in convolution layer. The
codes are publicly available at: https://github.com/huaweinoah/AdderNet.
