  This paper discusses a class of thresholding-based iterative selection
procedures (TISP) for model selection and shrinkage. People have long before
noticed the weakness of the convex $l_1$-constraint (or the soft-thresholding)
in wavelets and have designed many different forms of nonconvex penalties to
increase model sparsity and accuracy. But for a nonorthogonal regression
matrix, there is great difficulty in both investigating the performance in
theory and solving the problem in computation. TISP provides a simple and
efficient way to tackle this so that we successfully borrow the rich results in
the orthogonal design to solve the nonconvex penalized regression for a general
design matrix. Our starting point is, however, thresholding rules rather than
penalty functions. Indeed, there is a universal connection between them. But a
drawback of the latter is its non-unique form, and our approach greatly
facilitates the computation and the analysis. In fact, we are able to build the
convergence theorem and explore theoretical properties of the selection and
estimation via TISP nonasymptotically. More importantly, a novel Hybrid-TISP is
proposed based on hard-thresholding and ridge-thresholding. It provides a
fusion between the $l_0$-penalty and the $l_2$-penalty, and adaptively achieves
the right balance between shrinkage and selection in statistical modeling. In
practice, Hybrid-TISP shows superior performance in test-error and is
parsimonious.
