  Multi-species distribution modeling, which relates the occurrence of multiple
species to environmental variables, is an important tool used by ecologists for
both predicting the distribution of species in a community and identifying the
important variables driving species co-occurrences. Recently, Dunstan, Foster
and Darnell [Ecol. Model. 222 (2011) 955-963] proposed using finite mixture of
regression (FMR) models for multi-species distribution modeling, where species
are clustered based on their environmental response to form a small number of
"archetypal responses." As an illustrative example, they applied their mixture
model approach to a presence-absence data set of 200 marine organisms,
collected along the Great Barrier Reef in Australia. Little attention, however,
was given to the problem of model selection - since the archetypes (mixture
components) may depend on different but likely overlapping sets of covariates,
a method is needed for performing variable selection on all components
simultaneously. In this article, we consider using penalized likelihood
functions for variable selection in FMR models. We propose two penalties which
exploit the grouped structure of the covariates, that is, each covariate is
represented by a group of coefficients, one for each component. This leads to
an attractive form of shrinkage that allows a covariate to be removed from all
components simultaneously. Both penalties are shown to possess specific forms
of variable selection consistency, with simulations indicating they outperform
other methods which do not take into account the grouped structure. When
applied to the Great Barrier Reef data set, penalized FMR models offer more
insight into the important variables driving species co-occurrence in the
marine community (compared to previous results where no model selection was
conducted), while offering a computationally stable method of modeling complex
species-environment relationships (through regularization).
