  Data parallel training is widely used for scaling distributed deep neural
network (DNN) training. However, the performance benefits are often limited by
the communication-heavy parameter synchronization step. In this paper, we take
advantage of the domain specific knowledge of DNN training and overlap
parameter synchronization with computation in order to improve the training
performance. We make two key observations: (1) the optimal data representation
granularity for the communication may differ from that used by the underlying
DNN model implementation and (2) different parameters can afford different
synchronization delays. Based on these observations, we propose a new
synchronization mechanism called Priority-based Parameter Propagation (P3). P3
synchronizes parameters at a finer granularity and schedules data transmission
in such a way that the training process incurs minimal communication delay. We
show that P3 can improve the training throughput of ResNet-50, Sockeye and
VGG-19 by as much as 25%, 38% and 66% respectively on clusters with realistic
network bandwidth
