  The extent to which text-only language models (LMs) learn to represent
features of the non-linguistic world is an open question. Prior work has shown
that pretrained LMs can be taught to caption images when a vision model's
parameters are optimized to encode images in the language space. We test a
stronger hypothesis: that the conceptual representations learned by frozen
text-only models and vision-only models are similar enough that this can be
achieved with a linear map. We show that the image representations from vision
models can be transferred as continuous prompts to frozen LMs by training only
a single linear projection. Using these to prompt the LM achieves competitive
performance on captioning and visual question answering tasks compared to
models that tune both the image encoder and text decoder (such as the MAGMA
model). We compare three image encoders with increasing amounts of linguistic
supervision seen during pretraining: BEIT (no linguistic information),
NF-ResNET (lexical category information), and CLIP (full natural language
descriptions). We find that all three encoders perform equally well at
transferring visual property information to the language model (e.g., whether
an animal is large or small), but that image encoders pretrained with
linguistic supervision more saliently encode category information (e.g.,
distinguishing hippo vs. elephant) and thus perform significantly better on
benchmark language-and-vision tasks. Our results indicate that LMs encode
conceptual information structurally similarly to vision-based models, even
those that are solely trained on images. Code is available here:
https://github.com/jmerullo/limber
