  We propose BinaryRelax, a simple two-phase algorithm, for training deep
neural networks with quantized weights. The set constraint that characterizes
the quantization of weights is not imposed until the late stage of training,
and a sequence of \emph{pseudo} quantized weights is maintained. Specifically,
we relax the hard constraint into a continuous regularizer via Moreau envelope,
which turns out to be the squared Euclidean distance to the set of quantized
weights. The pseudo quantized weights are obtained by linearly interpolating
between the float weights and their quantizations. A continuation strategy is
adopted to push the weights towards the quantized state by gradually increasing
the regularization parameter. In the second phase, exact quantization scheme
with a small learning rate is invoked to guarantee fully quantized weights. We
test BinaryRelax on the benchmark CIFAR and ImageNet color image datasets to
demonstrate the superiority of the relaxed quantization approach and the
improved accuracy over the state-of-the-art training methods. Finally, we prove
the convergence of BinaryRelax under an approximate orthogonality condition.
