  There are limitations in learning language from text alone. Therefore, recent
focus has been on developing multimodal models. However, few benchmarks exist
that can measure what language models learn about language from multimodal
training. We hypothesize that training on a visual modality should improve on
the visual commonsense knowledge in language models. Therefore, we introduce
two evaluation tasks for measuring visual commonsense knowledge in language
models and use them to evaluate different multimodal models and unimodal
baselines. Primarily, we find that the visual commonsense knowledge is not
significantly different between the multimodal models and unimodal baseline
models trained on visual text data.
