  Distributed Optimization is an increasingly important subject area with the
rise of multi-agent control and optimization. We consider a decentralized
stochastic optimization problem where the agents on a graph aim to
asynchronously optimize a collective (additive) objective function consisting
of agents' individual (possibly non-convex) local objective functions. Each
agent only has access to a noisy estimate of the gradient of its own function
(one component of the sum of objective functions). We proposed an asynchronous
distributed algorithm for such a class of problems. The algorithm combines
stochastic gradients with tracking in an asynchronous push-sum framework and
obtain the standard sublinear convergence rate for general non-convex
functions, matching the rate of centralized stochastic gradient descent SGD.
  Our experiments on a non-convex image classification task using convolutional
neural network validate the convergence of our proposed algorithm across
different number of nodes and graph connectivity percentages.
