  We develop a distributed stochastic gradient descent algorithm for solving
non-convex optimization problems under the assumption that the local objective
functions are twice continuously differentiable with Lipschitz continuous
gradients and Hessians. We provide sufficient conditions on step-sizes that
guarantee the asymptotic mean-square convergence of the proposed algorithm. We
apply the developed algorithm to a distributed supervised-learning problem, in
which a set of networked agents collaboratively train their individual neural
nets to recognize handwritten digits in images. Results indicate that all
agents report similar performance that is also comparable to the performance of
a centrally trained neural net. Numerical results also show that the proposed
distributed algorithm allows the individual agents to recognize the digits even
though the training data corresponding to all the digits is not locally
available to each agent.
