  GANs are difficult to train due to convergence pathologies such as mode and
discriminator collapse. We introduce Lipizzaner, an open source software system
that allows machine learning engineers to train GANs in a distributed and
robust way. Lipizzaner distributes a competitive coevolutionary algorithm
which, by virtue of dual, adapting, generator and discriminator populations, is
robust to collapses. The algorithm is well suited to efficient distribution
because it uses a spatial grid abstraction. Training is local to each cell and
strong intermediate training results are exchanged among overlapping
neighborhoods allowing high performing solutions to propagate and improve with
more rounds of training. Experiments on common image datasets overcome critical
collapses. Communication overhead scales linearly when increasing the number of
compute instances and we observe that increasing scale leads to improved model
performance.
