  When training neural networks with low-precision computation, rounding errors
often cause stagnation or are detrimental to the convergence of the optimizers;
in this paper we study the influence of rounding errors on the convergence of
the gradient descent method for problems satisfying the Polyak-Lojasiewicz
inequality. Within this context, we show that, in contrast, biased stochastic
rounding errors may be beneficial since choosing a proper rounding strategy
eliminates the vanishing gradient problem and forces the rounding bias in a
descent direction. Furthermore, we obtain a bound on the convergence rate that
is stricter than the one achieved by unbiased stochastic rounding. The
theoretical analysis is validated by comparing the performances of various
rounding strategies when optimizing several examples using low-precision
fixed-point number formats.
