  In this paper we develop a nonparametric maximum likelihood estimate of the
mixing distribution of the parameters of a linear stochastic dynamical system.
This includes, for example, pharmacokinetic population models with process and
measurement noise that are linear in the state vector, input vector and the
process and measurement noise vectors. Most research in mixing distributions
only considers measurement noise. The advantages of the models with process
noise are that, in addition to the measurements errors, the uncertainties in
the model itself are taken into the account. For example, for deterministic
pharmacokinetic models, errors in dose amounts, administration times, and
timing of blood samples are typically not included. For linear stochastic
models, we use linear Kalman-Bucy filtering to calculate the likelihood of the
observations and then employ a nonparametric adaptive grid algorithm to find
the nonparametric maximum likelihood estimate of the mixing distribution. We
then use the directional derivatives of the estimated mixing distribution to
show that the result found attains a global maximum. A simple example using a
one compartment pharmacokinetic linear stochastic model is given. In addition
to population pharmacokinetics, this research also applies to empirical Bayes
estimation.
