  Binarization of neural networks is a dominant paradigm in neural networks
compression. The pioneering work BinaryConnect uses Straight Through Estimator
(STE) to mimic the gradients of the sign function, but it also causes the
crucial inconsistency problem. Most of the previous methods design different
estimators instead of STE to mitigate it. However, they ignore the fact that
when reducing the estimating error, the gradient stability will decrease
concomitantly. These highly divergent gradients will harm the model training
and increase the risk of gradient vanishing and gradient exploding. To fully
take the gradient stability into consideration, we present a new perspective to
the BNNs training, regarding it as the equilibrium between the estimating error
and the gradient stability. In this view, we firstly design two indicators to
quantitatively demonstrate the equilibrium phenomenon. In addition, in order to
balance the estimating error and the gradient stability well, we revise the
original straight through estimator and propose a power function based
estimator, Rectified Straight Through Estimator (ReSTE for short). Comparing to
other estimators, ReSTE is rational and capable of flexibly balancing the
estimating error with the gradient stability. Extensive experiments on CIFAR-10
and ImageNet datasets show that ReSTE has excellent performance and surpasses
the state-of-the-art methods without any auxiliary modules or losses.
