  We propose a new algorithm for training deep neural networks (DNNs) with
binary weights. In particular, we first cast the problem of training binary
neural networks (BiNNs) as a bilevel optimization instance and subsequently
construct flexible relaxations of this bilevel program. The resulting training
method shares its algorithmic simplicity with several existing approaches to
train BiNNs, in particular with the straight-through gradient estimator
successfully employed in BinaryConnect and subsequent methods. In fact, our
proposed method can be interpreted as an adaptive variant of the original
straight-through estimator that conditionally (but not always) acts like a
linear mapping in the backward pass of error propagation. Experimental results
demonstrate that our new algorithm offers favorable performance compared to
existing approaches.
