  We consider the problem of learning predictive models from longitudinal data,
consisting of irregularly repeated, sparse observations from a set of
individuals over time. Such data often exhibit {\em longitudinal correlation}
(LC) (correlations among observations for each individual over time), {\em
cluster correlation} (CC) (correlations among individuals that have similar
characteristics), or both. These correlations are often accounted for using
{\em mixed effects models} that include {\em fixed effects} and {\em random
effects}, where the fixed effects capture the regression parameters that are
shared by all individuals, whereas random effects capture those parameters that
vary across individuals. However, the current state-of-the-art methods are
unable to select the most predictive fixed effects and random effects from a
large number of variables, while accounting for complex correlation structure
in the data and non-linear interactions among the variables. We propose
Longitudinal Multi-Level Factorization Machine (LMLFM), to the best of our
knowledge, the first model to address these challenges in learning predictive
models from longitudinal data. We establish the convergence properties, and
analyze the computational complexity, of LMLFM. We present results of
experiments with both simulated and real-world longitudinal data which show
that LMLFM outperforms the state-of-the-art methods in terms of predictive
accuracy, variable selection ability, and scalability to data with large number
of variables. The code and supplemental material is available at
\url{https://github.com/junjieliang672/LMLFM}.
