  Recommendation systems are often trained with a tremendous amount of data,
and distributed training is the workhorse to shorten the training time. While
the training throughput can be increased by simply adding more workers, it is
also increasingly challenging to preserve the model quality. In this paper, we
present \shadowsync, a distributed framework specifically tailored to modern
scale recommendation system training. In contrast to previous works where
synchronization happens as part of the training process, \shadowsync separates
the synchronization from training and runs it in the background. Such isolation
significantly reduces the synchronization overhead and increases the
synchronization frequency, so that we are able to obtain both high throughput
and excellent model quality when training at scale. The superiority of our
procedure is confirmed by experiments on training deep neural networks for
click-through-rate prediction tasks. Our framework is capable to express data
parallelism and/or model parallelism, generic to host various types of
synchronization algorithms, and readily applicable to large scale problems in
other areas.
