  While distributed training significantly speeds up the training process of
the deep neural network (DNN), the utilization of the cluster is relatively low
due to the time-consuming data synchronizing between workers. To alleviate this
problem, a novel Hierarchical Parallel SGD (HPSGD) strategy is proposed based
on the observation that the data synchronization phase can be paralleled with
the local training phase (i.e., Feed-forward and back-propagation).
Furthermore, an improved model updating method is unitized to remedy the
introduced stale gradients problem, which commits updates to the replica (i.e.,
a temporary model that has the same parameters as the global model) and then
merges the average changes to the global model. Extensive experiments are
conducted to demonstrate that the proposed HPSGD approach substantially boosts
the distributed DNN training, reduces the disturbance of the stale gradients
and achieves better accuracy in given fixed wall-time.
