  Improved state space models, such as Recurrent State Space Models (RSSMs),
are a key factor behind recent advances in model-based reinforcement learning
(RL). Yet, despite their empirical success, many of the underlying design
choices are not well understood. We show that RSSMs use a suboptimal inference
scheme and that models trained using this inference overestimate the aleatoric
uncertainty of the ground truth system. We find this overestimation implicitly
regularizes RSSMs and allows them to succeed in model-based RL. We postulate
that this implicit regularization fulfills the same functionality as explicitly
modeling epistemic uncertainty, which is crucial for many other model-based RL
approaches. Yet, overestimating aleatoric uncertainty can also impair
performance in cases where accurately estimating it matters, e.g., when we have
to deal with occlusions, missing observations, or fusing sensor modalities at
different frequencies. Moreover, the implicit regularization is a side-effect
of the inference scheme and not the result of a rigorous, principled
formulation, which renders analyzing or improving RSSMs difficult. Thus, we
propose an alternative approach building on well-understood components for
modeling aleatoric and epistemic uncertainty, dubbed Variational Recurrent
Kalman Network (VRKN). This approach uses Kalman updates for exact smoothing
inference in a latent space and Monte Carlo Dropout to model epistemic
uncertainty. Due to the Kalman updates, the VRKN can naturally handle missing
observations or sensor fusion problems with varying numbers of observations per
time step. Our experiments show that using the VRKN instead of the RSSM
improves performance in tasks where appropriately capturing aleatoric
uncertainty is crucial while matching it in the deterministic standard
benchmarks.
