  The demand for artificial intelligence has grown significantly over the last
decade and this growth has been fueled by advances in machine learning
techniques and the ability to leverage hardware acceleration. However, in order
to increase the quality of predictions and render machine learning solutions
feasible for more complex applications, a substantial amount of training data
is required. Although small machine learning models can be trained with modest
amounts of data, the input for training larger models such as neural networks
grows exponentially with the number of parameters. Since the demand for
processing training data has outpaced the increase in computation power of
computing machinery, there is a need for distributing the machine learning
workload across multiple machines, and turning the centralized into a
distributed system. These distributed systems present new challenges, first and
foremost the efficient parallelization of the training process and the creation
of a coherent model. This article provides an extensive overview of the current
state-of-the-art in the field by outlining the challenges and opportunities of
distributed machine learning over conventional (centralized) machine learning,
discussing the techniques used for distributed machine learning, and providing
an overview of the systems that are available.
