  Machine learning models made up of millions or billions of parameters are
trained and served on large multi-GPU systems. As models grow in size and
execute on more GPUs, the collective communications used in these applications
become a bottleneck. Custom collective algorithms optimized for both particular
network topologies and application specific communication patterns can
alleviate this bottleneck and help these applications scale. However, correctly
and efficiently implementing custom algorithms is challenging.
  This paper introduces GC3, a system for programmable GPU communication. GC3
provides a domain specific language for writing collective communication
algorithms and an optimizing compiler for lowering them to an executable form,
which can be executed efficiently and flexibly in an interpreter based runtime.
We used GC3 to write novel collective algorithms for AllReduce and AllToAll
that are up to $1.9\times$ and $1.3\times$ faster than hand-optimized
implementations, respectively.
