  We propose a novel probabilistic generative model for action sequences. The
model is termed the Action Point Process VAE (APP-VAE), a variational
auto-encoder that can capture the distribution over the times and categories of
action sequences. Modeling the variety of possible action sequences is a
challenge, which we show can be addressed via the APP-VAE's use of latent
representations and non-linear functions to parameterize distributions over
which event is likely to occur next in a sequence and at what time. We
empirically validate the efficacy of APP-VAE for modeling action sequences on
the MultiTHUMOS and Breakfast datasets.
