  Adjusting batch sizes and adaptively tuning other hyperparameters can
significantly speed up deep neural network (DNN) training. Despite the ubiquity
of heterogeneous clusters, existing adaptive DNN training techniques solely
consider homogeneous environments. Optimizing distributed DNN training over
heterogeneous clusters is technically challenging, and directly adapting
existing techniques results in low utilization and poor performance. To solve
this problem, we introduce Cannikin -- a novel data-parallel distributed
training system. Cannikin achieves efficient and near-optimal performance by
accurately modeling the optimal system performance and predicting adaptive
batch size training metrics for DNNs in heterogeneous clusters. We implemented
Cannikin in PyTorch and conducted experiments over 16 GPUs in Chameleon.
Empirical results show that Cannikin reduces DNN training in heterogeneous
clusters by up to $52\%$ compared to the state-of-the-art adaptive training
system and up to $85\%$ compared to native PyTorch DistributedDataParallel.
