  Distributed training algorithms of deep neural networks show impressive
convergence speedup properties on very large problems. However, they inherently
suffer from communication related slowdowns and communication topology becomes
a crucial design choice. Common approaches supported by most machine learning
frameworks are: 1) Synchronous decentralized algorithms relying on a
peer-to-peer All Reduce topology that is sensitive to stragglers and
communication delays. 2) Asynchronous centralised algorithms with a server
based topology that is prone to communication bottleneck. Researchers also
suggested asynchronous decentralized algorithms designed to avoid the
bottleneck and speedup training, however, those commonly use inexact sparse
averaging that may lead to a degradation in accuracy. In this paper, we propose
Local Asynchronous SGD (LASGD), an asynchronous decentralized algorithm that
relies on All Reduce for model synchronization.
  We empirically validate LASGD's performance on image classification tasks on
the ImageNet dataset. Our experiments demonstrate that LASGD accelerates
training compared to SGD and state of the art gossip based approaches.
