  Deep Neural Network (DNN) models have continuously been growing in size in
order to improve the accuracy and quality of the models. Moreover, for training
of large DNN models, the use of heterogeneous GPUs is inevitable due to the
short release cycle of new GPU architectures. In this paper, we investigate how
to enable training of large DNN models on a heterogeneous GPU cluster that
possibly includes whimpy GPUs that, as a standalone, could not be used for
training. We present a DNN training system, HetPipe (Heterogeneous Pipeline),
that integrates pipelined model parallelism (PMP) with data parallelism (DP).
In HetPipe, a group of multiple GPUs, called a virtual worker, processes
minibatches in a pipelined manner, and multiple such virtual workers employ
data parallelism for higher performance. We also propose a novel parameter
synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to
accommodate both PMP and DP for virtual workers, and provide convergence proof
of WSP. Our experimental results on a given heterogeneous setting show that
with HetPipe, DNN models converge up to 49% faster compared to the
state-of-the-art DP technique.
