  Visual Question Answering (VQA) needs a means of evaluating the strengths and
weaknesses of models. One aspect of such an evaluation is the evaluation of
compositional generalisation, or the ability of a model to answer well on
scenes whose scene-setups are different from the training set. Therefore, for
this purpose, we need datasets whose train and test sets differ significantly
in composition. In this work, we present several quantitative measures of
compositional separation and find that popular datasets for VQA are not good
evaluators. To solve this, we present Uncommon Objects in Unseen Configurations
(UOUC), a synthetic dataset for VQA. UOUC is at once fairly complex while also
being well-separated, compositionally. The object-class of UOUC consists of 380
clasess taken from 528 characters from the Dungeons and Dragons game. The train
set of UOUC consists of 200,000 scenes; whereas the test set consists of 30,000
scenes. In order to study compositional generalisation, simple reasoning and
memorisation, each scene of UOUC is annotated with up to 10 novel questions.
These deal with spatial relationships, hypothetical changes to scenes,
counting, comparison, memorisation and memory-based reasoning. In total, UOUC
presents over 2 million questions. UOUC also finds itself as a strong challenge
to well-performing models for VQA. Our evaluation of recent models for VQA
shows poor compositional generalisation, and comparatively lower ability
towards simple reasoning. These results suggest that UOUC could lead to
advances in research by being a strong benchmark for VQA.
