  Nonlinear Mixed effects models are hidden variables models that are widely
used in many fields such as pharmacometrics. In such models, the distribution
characteristics of hidden variables can be specified by including several
parameters such as covariates or correlations which must be selected. Recent
development of pharmacogenomics has brought averaged/high dimensional problems
to the field of nonlinear mixed effects modeling for which standard covariates
selection techniques like stepwise methods are not well suited. The selection
of covariates and correlation parameters using a penalized likelihood approach
is proposed. The penalized likelihood problem is solved using a stochastic
proximal gradient algorithm to avoid inner-outer iterations. Speed of
convergence of the proximal gradient algorithm is improved using component-wise
adaptive gradient step sizes. The practical implementation and tuning of the
proximal gradient algorithm are explored using simulations. Calibration of
regularization parameters is performed by minimizing the Bayesian Information
Criterion using particle swarm optimization, a zero-order optimization
procedure. The use of warm restart and parallelization allowed computing time
to be reduced significantly . The performance of the proposed method compared
to the traditional grid search strategy is explored using simulated data.
Finally, an application to real data from two pharmacokinetics studies is
provided, one studying an antifibrinolytic and the other studying an
antibiotic.
