  We consider a finite mixture of regressions (FMR) model for high-dimensional
inhomogeneous data where the number of covariates may be much larger than
sample size. We propose an l1-penalized maximum likelihood estimator in an
appropriate parameterization. This kind of estimation belongs to a class of
problems where optimization and theory for non-convex functions is needed. This
distinguishes itself very clearly from high-dimensional estimation with convex
loss- or objective functions, as for example with the Lasso in linear or
generalized linear models. Mixture models represent a prime and important
example where non-convexity arises.
  For FMR models, we develop an efficient EM algorithm for numerical
optimization with provable convergence properties. Our penalized estimator is
numerically better posed (e.g., boundedness of the criterion function) than
unpenalized maximum likelihood estimation, and it allows for effective
statistical regularization including variable selection. We also present some
asymptotic theory and oracle inequalities: due to non-convexity of the negative
log-likelihood function, different mathematical arguments are needed than for
problems with convex losses. Finally, we apply the new method to both simulated
and real data.
