  Generalized linear mixed models (GLMMs) are widely used in research for their
ability to model correlated outcomes with non-Gaussian conditional
distributions. The proper selection of fixed and random effects is a critical
part of the modeling process since model misspecification may lead to
significant bias. However, the joint selection of fixed and random effects has
historically been limited to lower-dimensional GLMMs, largely due to the use of
criterion-based model selection strategies. Here we present the R package
glmmPen, one of the first to select fixed and random effects in higher
dimension using a penalized GLMM modeling framework. Model parameters are
estimated using a Monte Carlo Expectation Conditional Minimization (MCECM)
algorithm, which leverages Stan and RcppArmadillo for increased computational
efficiency. Our package supports the Binomial, Gaussian, and Poisson families
and multiple penalty functions. In this manuscript we discuss the modeling
procedure, estimation scheme, and software implementation through application
to a pancreatic cancer subtyping study. Simulation results show our method has
good performance in selecting both the fixed and random effects in high
dimensional GLMMs.
