  Inverse graphics -- the task of inverting an image into physical variables
that, when rendered, enable reproduction of the observed scene -- is a
fundamental challenge in computer vision and graphics. Disentangling an image
into its constituent elements, such as the shape, color, and material
properties of the objects of the 3D scene that produced it, requires a
comprehensive understanding of the environment. This requirement limits the
ability of existing carefully engineered approaches to generalize across
domains. Inspired by the zero-shot ability of large language models (LLMs) to
generalize to novel contexts, we investigate the possibility of leveraging the
broad world knowledge encoded in such models in solving inverse-graphics
problems. To this end, we propose the Inverse-Graphics Large Language Model
(IG-LLM), an inverse-graphics framework centered around an LLM, that
autoregressively decodes a visual embedding into a structured, compositional
3D-scene representation. We incorporate a frozen pre-trained visual encoder and
a continuous numeric head to enable end-to-end training. Through our
investigation, we demonstrate the potential of LLMs to facilitate inverse
graphics through next-token prediction, without the use of image-space
supervision. Our analysis opens up new possibilities for precise spatial
reasoning about images that exploit the visual knowledge of LLMs. We will
release our code and data to ensure the reproducibility of our investigation
and to facilitate future research at https://ig-llm.is.tue.mpg.de/
