  We present a parameterized synthetic dataset called Moving Symbols to support
the objective study of video prediction networks. Using several instantiations
of the dataset in which variation is explicitly controlled, we highlight issues
in an existing state-of-the-art approach and propose the use of a performance
metric with greater semantic meaning to improve experimental interpretability.
Our dataset provides canonical test cases that will help the community better
understand, and eventually improve, the representations learned by such
networks in the future. Code is available at
https://github.com/rszeto/moving-symbols .
