  In distributed machine learning, where agents collaboratively learn from
diverse private data sets, there is a fundamental tension between consensus and
optimality. In this paper, we build on recent algorithmic progresses in
distributed deep learning to explore various consensus-optimality trade-offs
over a fixed communication topology. First, we propose the incremental
consensus-based distributed SGD (i-CDSGD) algorithm, which involves multiple
consensus steps (where each agent communicates information with its neighbors)
within each SGD iteration. Second, we propose the generalized consensus-based
distributed SGD (g-CDSGD) algorithm that enables us to navigate the full
spectrum from complete consensus (all agents agree) to complete disagreement
(each agent converges to individual model parameters). We analytically
establish convergence of the proposed algorithms for strongly convex and
nonconvex objective functions; we also analyze the momentum variants of the
algorithms for the strongly convex case. We support our algorithms via
numerical experiments, and demonstrate significant improvements over existing
methods for collaborative deep learning.
