  This position paper argues for the use of \emph{structured generative models}
(SGMs) for scene understanding. This requires the reconstruction of a 3D scene
from an input image, whereby the contents of the image are causally explained
in terms of models of instantiated objects, each with their own type, shape,
appearance and pose, along with global variables like scene lighting and camera
parameters. This approach also requires scene models which account for the
co-occurrences and inter-relationships of objects in a scene. The SGM approach
has the merits that it is compositional and generative, which lead to
interpretability.
  To pursue the SGM agenda, we need models for objects and scenes, and
approaches to carry out inference. We first review models for objects, which
include ``things'' (object categories that have a well defined shape), and
``stuff'' (categories which have amorphous spatial extent). We then move on to
review \emph{scene models} which describe the inter-relationships of objects.
Perhaps the most challenging problem for SGMs is \emph{inference} of the
objects, lighting and camera parameters, and scene inter-relationships from
input consisting of a single or multiple images. We conclude with a discussion
of issues that need addressing to advance the SGM agenda.
