  Large language models (LLMs) have made significant advancements in natural
language understanding. However, through that enormous semantic representation
that the LLM has learnt, is it somehow possible for it to understand images as
well? This work investigates this question. To enable the LLM to process
images, we convert them into a representation given by Scalable Vector Graphics
(SVG). To study what the LLM can do with this XML-based textual description of
images, we test the LLM on three broad computer vision tasks: (i) visual
reasoning and question answering, (ii) image classification under distribution
shift, few-shot learning, and (iii) generating new images using visual
prompting. Even though we do not naturally associate LLMs with any visual
understanding capabilities, our results indicate that the LLM can often do a
decent job in many of these tasks, potentially opening new avenues for research
into LLMs' ability to understand image data. Our code, data, and models can be
found here https://github.com/mu-cai/svg-llm.
