  Many current methods to learn intuitive physics are based on interaction
networks and similar approaches. However, they rely on information that has
proven difficult to estimate directly from image data in the past. We aim to
narrow this gap by inferring all the semantic information needed from raw pixel
data in the form of a scene-graph. Our approach is based on neural-symbolic
capsules, which identify which objects in the scene are static, dynamic,
elastic or rigid, possible joints between them, as well as their collision
information. By integrating all this with interaction networks, we demonstrate
how our method is able to learn intuitive physics directly from image sequences
and apply its knowledge to new scenes and objects, resulting in an
inverse-simulation pipeline.
