  In order to extract the best possible performance from asynchronous
stochastic gradient descent one must increase the mini-batch size and scale the
learning rate accordingly. In order to achieve further speedup we introduce a
technique that delays gradient updates effectively increasing the mini-batch
size. Unfortunately with the increase of mini-batch size we worsen the stale
gradient problem in asynchronous stochastic gradient descent (SGD) which makes
the model convergence poor. We introduce local optimizers which mitigate the
stale gradient problem and together with fine tuning our momentum we are able
to train a shallow machine translation system 27% faster than an optimized
baseline with negligible penalty in BLEU.
