  This paper proposes a novel Stochastic Split Linearized Bregman Iteration
($S^{2}$-LBI) algorithm to efficiently train the deep network. The $S^{2}$-LBI
introduces an iterative regularization path with structural sparsity. Our
$S^{2}$-LBI combines the computational efficiency of the LBI, and model
selection consistency in learning the structural sparsity. The computed
solution path intrinsically enables us to enlarge or simplify a network, which
theoretically, is benefited from the dynamics property of our $S^{2}$-LBI
algorithm. The experimental results validate our $S^{2}$-LBI on MNIST and
CIFAR-10 dataset. For example, in MNIST, we can either boost a network with
only 1.5K parameters (1 convolutional layer of 5 filters, and 1 FC layer),
achieves 98.40\% recognition accuracy; or we simplify $82.5\%$ of parameters in
LeNet-5 network, and still achieves the 98.47\% recognition accuracy. In
addition, we also have the learning results on ImageNet, which will be added in
the next version of our report.
