  In high-dimensional data analysis, regularization methods pursuing sparsity
and/or low rank have received a lot of attention recently. To provide a proper
amount of shrinkage, it is typical to use a grid search and a model comparison
criterion to find the optimal regularization parameters. However, we show that
fixing the parameters across all folds may result in an inconsistency issue,
and it is more appropriate to cross-validate projection-selection patterns to
obtain the best coefficient estimate. Our in-sample error studies in jointly
sparse and rank-deficient models lead to a new class of information criteria
with four scale-free forms to bypass the estimation of the noise level. By use
of an identity, we propose a novel scale-free calibration to help
cross-validation achieve the minimax optimal error rate non-asymptotically.
Experiments support the efficacy of the proposed methods.
