  We introduce Variational State-Space Filters (VSSF), a new method for
unsupervised learning, identification, and filtering of latent Markov state
space models from raw pixels. We present a theoretically sound framework for
latent state space inference under heterogeneous sensor configurations. The
resulting model can integrate an arbitrary subset of the sensor measurements
used during training, enabling the learning of semi-supervised state
representations, thus enforcing that certain components of the learned latent
state space to agree with interpretable measurements. From this framework we
derive L-VSSF, an explicit instantiation of this model with linear latent
dynamics and Gaussian distribution parameterizations. We experimentally
demonstrate L-VSSF's ability to filter in latent space beyond the sequence
length of the training dataset across several different test environments.
