  We propose a general way to integrate procedural knowledge of a domain into
deep learning models. We apply it to the case of video prediction, building on
top of object-centric deep models and show that this leads to a better
performance than using data-driven models alone. We develop an architecture
that facilitates latent space disentanglement in order to use the integrated
procedural knowledge, and establish a setup that allows the model to learn the
procedural interface in the latent space using the downstream task of video
prediction. We contrast the performance to a state-of-the-art data-driven
approach and show that problems where purely data-driven approaches struggle
can be handled by using knowledge about the domain, providing an alternative to
simply collecting more data.
