  System identification of complex and nonlinear systems is a central problem
for model predictive control and model-based reinforcement learning. Despite
their complexity, such systems can often be approximated well by a set of
linear dynamical systems if broken into appropriate subsequences. This
mechanism not only helps us find good approximations of dynamics, but also
gives us deeper insight into the underlying system. Leveraging Bayesian
inference, Variational Autoencoders and Concrete relaxations, we show how to
learn a richer and more meaningful state space, e.g. encoding joint constraints
and collisions with walls in a maze, from partial and high-dimensional
observations. This representation translates into a gain of accuracy of learned
dynamics showcased on various simulated tasks.
