  In order to build agents with a rich understanding of their environment, one
key objective is to endow them with a grasp of intuitive physics; an ability to
reason about three-dimensional objects, their dynamic interactions, and
responses to forces. While some work on this problem has taken the approach of
building in components such as ready-made physics engines, other research aims
to extract general physical concepts directly from sensory data. In the latter
case, one challenge that arises is evaluating the learning system. Research on
intuitive physics knowledge in children has long employed a violation of
expectations (VOE) method to assess children's mastery of specific physical
concepts. We take the novel step of applying this method to artificial learning
systems. In addition to introducing the VOE technique, we describe a set of
probe datasets inspired by classic test stimuli from developmental psychology.
We test a baseline deep learning system on this battery, as well as on a
physics learning dataset ("IntPhys") recently posed by another research group.
Our results show how the VOE technique may provide a useful tool for tracking
physics knowledge in future research.
