  Model parameter synchronization across GPUs introduces high overheads for
data-parallel training at scale. Existing parameter synchronization protocols
cannot effectively leverage available network resources in the face of ever
increasing hardware heterogeneity. To address this, we propose Blink, a
collective communication library that dynamically generates optimal
communication primitives by packing spanning trees. We propose techniques to
minimize the number of trees generated and extend Blink to leverage
heterogeneous communication channels for faster data transfers. Evaluations
show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8x
faster model synchronization, and reduce end-to-end training time for image
classification tasks by up to 40%.
