  Sparse neural networks are highly desirable in deep learning in reducing its
complexity. The goal of this paper is to study how choices of regularization
parameters influence the sparsity level of learned neural networks. We first
derive the $\ell_1$-norm sparsity-promoting deep learning models including
single and multiple regularization parameters models, from a statistical
viewpoint. We then characterize the sparsity level of a regularized neural
network in terms of the choice of the regularization parameters. Based on the
characterizations, we develop iterative algorithms for selecting regularization
parameters so that the weight parameters of the resulting deep neural network
enjoy prescribed sparsity levels. Numerical experiments are presented to
demonstrate the effectiveness of the proposed algorithms in choosing desirable
regularization parameters and obtaining corresponding neural networks having
both of predetermined sparsity levels and satisfactory approximation accuracy.
