  Long training times for high-accuracy deep neural networks (DNNs) impede
research into new DNN architectures and slow the development of high-accuracy
DNNs. In this paper we present FireCaffe, which successfully scales deep neural
network training across a cluster of GPUs. We also present a number of best
practices to aid in comparing advancements in methods for scaling and
accelerating the training of deep neural networks. The speed and scalability of
distributed algorithms is almost always limited by the overhead of
communicating between servers; DNN training is not an exception to this rule.
Therefore, the key consideration here is to reduce communication overhead
wherever possible, while not degrading the accuracy of the DNN models that we
train. Our approach has three key pillars. First, we select network hardware
that achieves high bandwidth between GPU servers -- Infiniband or Cray
interconnects are ideal for this. Second, we consider a number of communication
algorithms, and we find that reduction trees are more efficient and scalable
than the traditional parameter server approach. Third, we optionally increase
the batch size to reduce the total quantity of communication during DNN
training, and we identify hyperparameters that allow us to reproduce the
small-batch accuracy while training with large batch sizes. When training
GoogLeNet and Network-in-Network on ImageNet, we achieve a 47x and 39x speedup,
respectively, when training on a cluster of 128 GPUs.
