  In computer vision and machine learning, a crucial challenge is to lower the
computation and memory demands for neural network inference. A commonplace
solution to address this challenge is through the use of binarization. By
binarizing the network weights and activations, one can significantly reduce
computational complexity by substituting the computationally expensive floating
operations with faster bitwise operations. This leads to a more efficient
neural network inference that can be deployed on low-resource devices. In this
work, we extend previous approaches that trained networks with discrete weights
using the local reparameterization trick to also allow for discrete
activations. The original approach optimized a distribution over the discrete
weights and uses the central limit theorem to approximate the pre-activation
with a continuous Gaussian distribution. Here we show that the probabilistic
modeling can also allow effective training of networks with discrete activation
as well. This further reduces runtime and memory footprint at inference time
with state-of-the-art results for networks with binary activations.
