  Combining the visual modality with pretrained language models has been
surprisingly effective for simple descriptive tasks such as image captioning.
More general text generation however remains elusive. We take a step back and
ask: How do these models work for more complex generative tasks, i.e.
conditioning on both text and images? Are multimodal models simply visually
adapted language models, or do they combine they reason jointly over
modalities?
  We investigate these questions in the context of self-rationalization
(jointly generating task labels/answers and free-text explanations) of three
tasks: (i) visual question answering in VQA-X, (ii) visual commonsense
reasoning in VCR, and (iii) visual-textual entailment in e-SNLI-VE. We show
that recent unimodal advances, CLIP image representations and scaling of
language models, do not consistently improve self-rationalization in multimodal
tasks. We find that no single model type works universally best across tasks,
datasets, and finetuning data sizes. Our findings motivate the need for novel
general backbones approach that move text generation from images and text
beyond image captioning.
