  Distributed deep learning systems (DDLS) train deep neural network models by
utilizing the distributed resources of a cluster. Developers of DDLS are
required to make many decisions to process their particular workloads in their
chosen environment efficiently. The advent of GPU-based deep learning, the
ever-increasing size of datasets and deep neural network models, in combination
with the bandwidth constraints that exist in cluster environments require
developers of DDLS to be innovative in order to train high quality models
quickly. Comparing DDLS side-by-side is difficult due to their extensive
feature lists and architectural deviations. We aim to shine some light on the
fundamental principles that are at work when training deep neural networks in a
cluster of independent machines by analyzing the general properties associated
with training deep learning models and how such workloads can be distributed in
a cluster to achieve collaborative model training. Thereby we provide an
overview of the different techniques that are used by contemporary DDLS and
discuss their influence and implications on the training process. To
conceptualize and compare DDLS, we group different techniques into categories,
thus establishing a taxonomy of distributed deep learning systems.
