  This paper proposes a theoretical framework to evaluate and compare the
performance of gradient-descent algorithms for distributed learning in relation
to their behavior around local minima in nonconvex environments. Previous works
have noticed that convergence toward flat local minima tend to enhance the
generalization ability of learning algorithms. This work discovers two
interesting results. First, it shows that decentralized learning strategies are
able to escape faster away from local minimizers and favor convergence toward
flatter minima relative to the centralized solution in the large-batch training
regime. Second, and importantly, the ultimate classification accuracy is not
solely dependent on the flatness of the local minimizer but also on how well a
learning algorithm can approach that minimum. In other words, the
classification accuracy is a function of both flatness and optimization
performance. The paper examines the interplay between the two measures of
flatness and optimization error closely. One important conclusion is that
decentralized strategies of the diffusion type deliver enhanced classification
accuracy because it strikes a more favorable balance between flatness and
optimization performance.
