  In prior works, stochastic dual coordinate ascent (SDCA) has been
parallelized in a multi-core environment where the cores communicate through
shared memory, or in a multi-processor distributed memory environment where the
processors communicate through message passing. In this paper, we propose a
hybrid SDCA framework for multi-core clusters, the most common high performance
computing environment that consists of multiple nodes each having multiple
cores and its own shared memory. We distribute data across nodes where each
node solves a local problem in an asynchronous parallel fashion on its cores,
and then the local updates are aggregated via an asynchronous across-node
update scheme. The proposed double asynchronous method converges to a global
solution for $L$-Lipschitz continuous loss functions, and at a linear
convergence rate if a smooth convex loss function is used. Extensive empirical
comparison has shown that our algorithm scales better than the best known
shared-memory methods and runs faster than previous distributed-memory methods.
Big datasets, such as one of 280 GB from the LIBSVM repository, cannot be
accommodated on a single node and hence cannot be solved by a parallel
algorithm. For such a dataset, our hybrid algorithm takes 30 seconds to achieve
a duality gap of $10^{-6}$ on 16 nodes each using 8 cores, which is
significantly faster than the best known distributed algorithms, such as
CoCoA+, that take more than 300 seconds on 16 nodes.
