  Modern high-dimensional methods often adopt the "bet on sparsity" principle,
while in supervised multivariate learning statisticians may face "dense"
problems with a large number of nonzero coefficients. This paper proposes a
novel clustered reduced-rank learning (CRL) framework that imposes two joint
matrix regularizations to automatically group the features in constructing
predictive factors. CRL is more interpretable than low-rank modeling and
relaxes the stringent sparsity assumption in variable selection. In this paper,
new information-theoretical limits are presented to reveal the intrinsic cost
of seeking for clusters, as well as the blessing from dimensionality in
multivariate learning. Moreover, an efficient optimization algorithm is
developed, which performs subspace learning and clustering with guaranteed
convergence. The obtained fixed-point estimators, though not necessarily
globally optimal, enjoy the desired statistical accuracy beyond the standard
likelihood setup under some regularity conditions. Moreover, a new kind of
information criterion, as well as its scale-free form, is proposed for cluster
and rank selection, and has a rigorous theoretical support without assuming an
infinite sample size. Extensive simulations and real-data experiments
demonstrate the statistical accuracy and interpretability of the proposed
method.
