  We consider the problem of predicting several response variables using the
same set of explanatory variables. This setting naturally induces a group
structure over the coefficient matrix, in which every explanatory variable
corresponds to a set of related coefficients. Most of the existing methods that
utilize this group formation assume that the similarities between related
coefficients arise solely through a joint sparsity structure. In this paper, we
propose a procedure for constructing an estimator of a multivariate regression
coefficient matrix that directly models and captures the within-group
similarities, by employing a multivariate linear mixed model formulation, with
a joint estimation of covariance matrices for coefficients and errors via
penalized likelihood. Our approach, which we term Multivariate random
Regression with Covariance Estimation (MrRCE) encourages structured similarity
in parameters, in which coefficients for the same variable in related tasks
sharing the same sign and similar magnitude. We illustrate the benefits of our
approach in synthetic and real examples, and show that the proposed method
outperforms natural competitors and alternative estimators under several model
settings.
