  Numerous visio-linguistic (V+L) representation learning methods have been
developed, yet existing datasets do not adequately evaluate the extent to which
they represent visual and linguistic concepts in a unified space. We propose
several novel evaluation settings for V+L models, including cross-modal
transfer. Furthermore, existing V+L benchmarks often report global accuracy
scores on the entire dataset, making it difficult to pinpoint the specific
reasoning tasks that models fail and succeed at. We present TraVLR, a synthetic
dataset comprising four V+L reasoning tasks. TraVLR's synthetic nature allows
us to constrain its training and testing distributions along task-relevant
dimensions, enabling the evaluation of out-of-distribution generalisation. Each
example in TraVLR redundantly encodes the scene in two modalities, allowing
either to be dropped or added during training or testing without losing
relevant information. We compare the performance of four state-of-the-art V+L
models, finding that while they perform well on test examples from the same
modality, they all fail at cross-modal transfer and have limited success
accommodating the addition or deletion of one modality. We release TraVLR as an
open challenge for the research community.
