  In this paper, we propose a Distributed Accumulated Newton Conjugate gradiEnt
(DANCE) method in which sample size is gradually increasing to quickly obtain a
solution whose empirical loss is under satisfactory statistical accuracy. Our
proposed method is multistage in which the solution of a stage serves as a warm
start for the next stage which contains more samples (including the samples in
the previous stage). The proposed multistage algorithm reduces the number of
passes over data to achieve the statistical accuracy of the full training set.
Moreover, our algorithm in nature is easy to be distributed and shares the
strong scaling property indicating that acceleration is always expected by
using more computing nodes. Various iteration complexity results regarding
descent direction computation, communication efficiency and stopping criteria
are analyzed under convex setting. Our numerical results illustrate that the
proposed method outperforms other comparable methods for solving learning
problems including neural networks.
