  Training the deep convolutional neural network for computer vision problems
is slow and inefficient, especially when it is large and distributed across
multiple devices. The inefficiency is caused by the backpropagation algorithm's
forward locking, backward locking, and update locking problems. Existing
solutions for acceleration either can only handle one locking problem or lead
to severe accuracy loss or memory inefficiency. Moreover, none of them consider
the straggler problem among devices. In this paper, we propose Layer-wise
Staleness and a novel efficient training algorithm, Diversely Stale Parameters
(DSP), to address these challenges. We also analyze the convergence of DSP with
two popular gradient-based methods and prove that both of them are guaranteed
to converge to critical points for non-convex problems. Finally, extensive
experimental results on training deep learning models demonstrate that our
proposed DSP algorithm can achieve significant training speedup with stronger
robustness than compared methods.
