  Latent dynamics discovery is challenging in extracting complex dynamics from
high-dimensional noisy neural data. Many dimensionality reduction methods have
been widely adopted to extract low-dimensional, smooth and time-evolving latent
trajectories. However, simple state transition structures, linear embedding
assumptions, or inflexible inference networks impede the accurate recovery of
dynamic portraits. In this paper, we propose a novel latent dynamic model that
is capable of capturing nonlinear, non-Markovian, long short-term
time-dependent dynamics via recurrent neural networks and tackling complex
nonlinear embedding via non-parametric Gaussian process. Due to the complexity
and intractability of the model and its inference, we also provide a powerful
inference network with bi-directional long short-term memory networks that
encode both past and future information into posterior distributions. In the
experiment, we show that our model outperforms other state-of-the-art methods
in reconstructing insightful latent dynamics from both simulated and
experimental neural datasets with either Gaussian or Poisson observations,
especially in the low-sample scenario. Our codes and additional materials are
available at https://github.com/sheqi/GP-RNN_UAI2019.
