  We consider the problem of training a machine learning model over a network
of nodes in a fully decentralized framework. The nodes take a Bayesian-like
approach via the introduction of a belief over the model parameter space. We
propose a distributed learning algorithm in which nodes update their belief by
aggregate information from their one-hop neighbors to learn a model that best
fits the observations over the entire network. In addition, we also obtain
sufficient conditions to ensure that the probability of error is small for
every node in the network. We discuss approximations required for applying this
algorithm to train Deep Neural Networks (DNNs). Experiments on training linear
regression model and on training a DNN show that the proposed learning rule
algorithm provides a significant improvement in the accuracy compared to the
case where nodes learn without cooperation.
