  Supervised linear feature extraction can be achieved by fitting a reduced
rank multivariate model. This paper studies rank penalized and rank constrained
vector generalized linear models. From the perspective of thresholding rules,
we build a framework for fitting singular value penalized models and use it for
feature extraction. Through solving the rank constraint form of the problem, we
propose progressive feature space reduction for fast computation in high
dimensions with little performance loss. A novel projective cross-validation is
proposed for parameter tuning in such nonconvex setups. Real data applications
are given to show the power of the methodology in supervised dimension
reduction and feature extraction.
