  Alternating Direction Method of Multipliers (ADMM) has recently been proposed
as a potential alternative optimizer to the Stochastic Gradient Descent(SGD)
for deep learning problems. This is because ADMM can solve gradient vanishing
and poor conditioning problems. Moreover, it has shown good scalability in many
large-scale deep learning applications. However, there still lacks a parallel
ADMM computational framework for deep neural networks because of layer
dependency among variables. In this paper, we propose a novel parallel deep
learning ADMM framework (pdADMM) to achieve layer parallelism: parameters in
each layer of neural networks can be updated independently in parallel. The
convergence of the proposed pdADMM to a critical point is theoretically proven
under mild conditions. The convergence rate of the pdADMM is proven to be
$o(1/k)$ where $k$ is the number of iterations. Extensive experiments on six
benchmark datasets demonstrated that our proposed pdADMM can lead to more than
10 times speedup for training large-scale deep neural networks, and
outperformed most of the comparison methods. Our code is available at:
https://github.com/xianggebenben/pdADMM.
