  In this paper, we develop a new algorithm, Annealed Skewed SGD - AskewSGD -
for training deep neural networks (DNNs) with quantized weights. First, we
formulate the training of quantized neural networks (QNNs) as a smoothed
sequence of interval-constrained optimization problems. Then, we propose a new
first-order stochastic method, AskewSGD, to solve each constrained optimization
subproblem. Unlike algorithms with active sets and feasible directions,
AskewSGD avoids projections or optimization under the entire feasible set and
allows iterates that are infeasible. The numerical complexity of AskewSGD is
comparable to existing approaches for training QNNs, such as the
straight-through gradient estimator used in BinaryConnect, or other state of
the art methods (ProxQuant, LUQ). We establish convergence guarantees for
AskewSGD (under general assumptions for the objective function). Experimental
results show that the AskewSGD algorithm performs better than or on par with
state of the art methods in classical benchmarks.
