  We discuss problems with the standard approaches to evaluation for tasks like
visual question answering, and argue that artificial data can be used to
address these as a complement to current practice. We demonstrate that with the
help of existing 'deep' linguistic processing technology we are able to create
challenging abstract datasets, which enable us to investigate the language
understanding abilities of multimodal deep learning models in detail, as
compared to a single performance value on a static and monolithic dataset.
