  The ability for computational agents to reason about the high-level content
of real world scene images is important for many applications. Existing
attempts at addressing the problem of complex scene understanding lack
representational power, efficiency, and the ability to create robust
meta-knowledge about scenes. In this paper, we introduce scenarios as a new way
of representing scenes. The scenario is a simple, low-dimensional, data-driven
representation consisting of sets of frequently co-occurring objects and is
useful for a wide range of scene understanding tasks. We learn scenarios from
data using a novel matrix factorization method which we integrate into a new
neural network architecture, the ScenarioNet. Using ScenarioNet, we can recover
semantic information about real world scene images at three levels of
granularity: 1) scene categories, 2) scenarios, and 3) objects. Training a
single ScenarioNet model enables us to perform scene classification, scenario
recognition, multi-object recognition, content-based scene image retrieval, and
content-based image comparison. In addition to solving many tasks in a single,
unified framework, ScenarioNet is more computationally efficient than other
CNNs because it requires significantly fewer parameters while achieving similar
performance on benchmark tasks and is more interpretable because it produces
explanations when making decisions. We validate the utility of scenarios and
ScenarioNet on a diverse set of scene understanding tasks on several benchmark
datasets.
