  Dynamics model learning deals with the task of inferring unknown dynamics
from measurement data and predicting the future behavior of the system. A
typical approach to address this problem is to train recurrent models. However,
predictions with these models are often not physically meaningful. Further,
they suffer from deteriorated behavior over time due to accumulating errors.
Often, simulators building on first principles are available being physically
meaningful by design. However, modeling simplifications typically cause
inaccuracies in these models. Consequently, hybrid modeling is an emerging
trend that aims to combine the best of both worlds. In this paper, we propose a
new approach to hybrid modeling, where we inform the latent states of a learned
model via a black-box simulator. This allows to control the predictions via the
simulator preventing them from accumulating errors. This is especially
challenging since, in contrast to previous approaches, access to the
simulator's latent states is not available. We tackle the task by leveraging
observers, a well-known concept from control theory, inferring unknown latent
states from observations and dynamics over time. In our learning-based setting,
we jointly learn the dynamics and an observer that infers the latent states via
the simulator. Thus, the simulator constantly corrects the latent states,
compensating for modeling mismatch caused by learning. To maintain flexibility,
we train an RNN-based residuum for the latent states that cannot be informed by
the simulator.
