  A large fraction of the arithmetic operations required to evaluate deep
neural networks (DNNs) consists of matrix multiplications, in both convolution
and fully connected layers. We perform end-to-end learning of low-cost
approximations of matrix multiplications in DNN layers by casting matrix
multiplications as 2-layer sum-product networks (SPNs) (arithmetic circuits)
and learning their (ternary) edge weights from data. The SPNs disentangle
multiplication and addition operations and enable us to impose a budget on the
number of multiplication operations. Combining our method with knowledge
distillation and applying it to image classification DNNs (trained on ImageNet)
and language modeling DNNs (using LSTMs), we obtain a first-of-a-kind reduction
in number of multiplications (over 99.5%) while maintaining the predictive
performance of the full-precision models. Finally, we demonstrate that the
proposed framework is able to rediscover Strassen's matrix multiplication
algorithm, learning to multiply $2 \times 2$ matrices using only 7
multiplications instead of 8.
