  $\ell_1$ based sparse regularization plays a central role in compressive
sensing and image processing. In this paper, we propose $\ell_1$DecNet, as an
unfolded network derived from a variational decomposition model incorporating
$\ell_1$ related sparse regularization and solved by scaled alternating
direction method of multipliers (ADMM). $\ell_1$DecNet effectively decomposes
an input image into a sparse feature and a learned dense feature, and thus
helps the subsequent sparse feature related operations. Based on this, we
develop $\ell_1$DecNet+, a learnable architecture framework consisting of our
$\ell_1$DecNet and a segmentation module which operates over extracted sparse
features instead of original images. This architecture combines well the
benefits of mathematical modeling and data-driven approaches. To our best
knowledge, this is the first study to incorporate mathematical image prior into
feature extraction in segmentation network structures. Moreover, our
$\ell_1$DecNet+ framework can be easily extended to 3D case. We evaluate the
effectiveness of $\ell_1$DecNet+ on two commonly encountered sparse
segmentation tasks: retinal vessel segmentation in medical image processing and
pavement crack detection in industrial abnormality identification. Experimental
results on different datasets demonstrate that, our $\ell_1$DecNet+
architecture with various lightweight segmentation modules can achieve equal or
better performance than their enlarged versions respectively. This leads to
especially practical advantages on resource-limited devices.
