  Most work in the deep learning systems community has focused on faster
inference, but arriving at a trained model requires lengthy experiments.
Accelerating training lets developers iterate faster and come up with better
models. DNN training is often seen as a compute-bound problem, best done in a
single large compute node with many GPUs. As DNNs get bigger, training requires
going distributed. Distributed deep neural network (DDNN) training constitutes
an important workload on the cloud. Larger DNN models and faster compute
engines shift the training performance bottleneck from computation to
communication. Our experiments show existing DNN training frameworks do not
scale in a typical cloud environment due to insufficient bandwidth and
inefficient parameter server software stacks.We propose PBox, a balanced,
scalable central PS hardware that balances compute and communication resources,
and PHub, a high performance parameter server (PS) software design that
provides an optimized network stack and a streamlined gradient processing
pipeline to benefit common PS setups to utilize PBox. We show that in a typical
cloud environment, PBox can achieve up to 3.8x speedup over state-of-the-art
designs when training ImageNet. We discuss future directions of integrating
PBox with programmable switches for in-network aggregation during training,
leveraging the datacenter network topology to reduce bandwidth usage and
localize data movement.
