  The study of sparsity in Convolutional Neural Networks (CNNs) has become
widespread to compress and accelerate models in environments with limited
resources. By constraining N consecutive weights along the output channel to be
group-wise non-zero, the recent network with 1$\times$N sparsity has received
tremendous popularity for its three outstanding advantages: 1) A large amount
of storage space saving by a \emph{Block Sparse Row} matrix. 2) Excellent
performance at a high sparsity. 3) Significant speedups on CPUs with Advanced
Vector Extensions. Recent work requires selecting and fine-tuning 1$\times$N
sparse weights based on dense pre-trained weights, leading to the problems such
as expensive training cost and memory access, sub-optimal model quality, as
well as unbalanced workload across threads (different sparsity across output
channels). To overcome them, this paper proposes a novel \emph{\textbf{S}oft
\textbf{U}niform \textbf{B}lock \textbf{P}runing} (SUBP) approach to train a
uniform 1$\times$N sparse structured network from scratch. Specifically, our
approach tends to repeatedly allow pruned blocks to regrow to the network based
on block angular redundancy and importance sampling in a uniform manner
throughout the training process. It not only makes the model less dependent on
pre-training, reduces the model redundancy and the risk of pruning the
important blocks permanently but also achieves balanced workload. Empirically,
on ImageNet, comprehensive experiments across various CNN architectures show
that our SUBP consistently outperforms existing 1$\times$N and structured
sparsity methods based on pre-trained models or training from scratch. Source
codes and models are available at \url{https://github.com/JingyangXiang/SUBP}.
