  Unsupervised learning plays an important role in many fields, such as
artificial intelligence, machine learning, and neuroscience. Compared to static
data, methods for extracting low-dimensional structure for dynamic data are
lagging. We developed a novel information-theoretic framework, Compressed
Predictive Information Coding (CPIC), to extract useful representations from
dynamic data. CPIC selectively projects the past (input) into a linear subspace
that is predictive about the compressed data projected from the future
(output). The key insight of our framework is to learn representations by
minimizing the compression complexity and maximizing the predictive information
in latent space. We derive variational bounds of the CPIC loss which induces
the latent space to capture information that is maximally predictive. Our
variational bounds are tractable by leveraging bounds of mutual information. We
find that introducing stochasticity in the encoder robustly contributes to
better representation. Furthermore, variational approaches perform better in
mutual information estimation compared with estimates under a Gaussian
assumption. We demonstrate that CPIC is able to recover the latent space of
noisy dynamical systems with low signal-to-noise ratios, and extracts features
predictive of exogenous variables in neuroscience data.
