  Scene understanding includes many related sub-tasks, such as scene
categorization, depth estimation, object detection, etc. Each of these
sub-tasks is often notoriously hard, and state-of-the-art classifiers already
exist for many of them. These classifiers operate on the same raw image and
provide correlated outputs. It is desirable to have an algorithm that can
capture such correlation without requiring any changes to the inner workings of
any classifier.
  We propose Feedback Enabled Cascaded Classification Models (FE-CCM), that
jointly optimizes all the sub-tasks, while requiring only a `black-box'
interface to the original classifier for each sub-task. We use a two-layer
cascade of classifiers, which are repeated instantiations of the original ones,
with the output of the first layer fed into the second layer as input. Our
training method involves a feedback step that allows later classifiers to
provide earlier classifiers information about which error modes to focus on. We
show that our method significantly improves performance in all the sub-tasks in
the domain of scene understanding, where we consider depth estimation, scene
categorization, event categorization, object detection, geometric labeling and
saliency detection. Our method also improves performance in two robotic
applications: an object-grasping robot and an object-finding robot.
