  Optimization of Binarized Neural Networks (BNNs) currently relies on
real-valued latent weights to accumulate small update steps. In this paper, we
argue that these latent weights cannot be treated analogously to weights in
real-valued networks. Instead their main role is to provide inertia during
training. We interpret current methods in terms of inertia and provide novel
insights into the optimization of BNNs. We subsequently introduce the first
optimizer specifically designed for BNNs, Binary Optimizer (Bop), and
demonstrate its performance on CIFAR-10 and ImageNet. Together, the
redefinition of latent weights as inertia and the introduction of Bop enable a
better understanding of BNN optimization and open up the way for further
improvements in training methodologies for BNNs. Code is available at:
https://github.com/plumerai/rethinking-bnn-optimization
