  Deep learning models (DLMs) are state-of-the-art techniques in speech
recognition. However, training good DLMs can be time consuming especially for
production-size models and corpora. Although several parallel training
algorithms have been proposed to improve training efficiency, there is no clear
guidance on which one to choose for the task in hand due to lack of systematic
and fair comparison among them. In this paper we aim at filling this gap by
comparing four popular parallel training algorithms in speech recognition,
namely asynchronous stochastic gradient descent (ASGD), blockwise model-update
filtering (BMUF), bulk synchronous parallel (BSP) and elastic averaging
stochastic gradient descent (EASGD), on 1000-hour LibriSpeech corpora using
feed-forward deep neural networks (DNNs) and convolutional, long short-term
memory, DNNs (CLDNNs). Based on our experiments, we recommend using BMUF as the
top choice to train acoustic models since it is most stable, scales well with
number of GPUs, can achieve reproducible results, and in many cases even
outperforms single-GPU SGD. ASGD can be used as a substitute in some cases.
