  Neural networks have shown to be a practical way of building a very complex
mapping between a pre-specified input space and output space. For example, a
convolutional neural network (CNN) mapping an image into one of a thousand
object labels is approaching human performance in this particular task. However
the mapping (neural network) does not automatically lend itself to other forms
of queries, for example, to detect/reconstruct object instances, to enforce
top-down signal on ambiguous inputs, or to recover object instances from
occlusion. One way to address these queries is a backward pass through the
network that fuses top-down and bottom-up information. In this paper, we show a
way of building such a backward pass by defining a generative model of the
neural network's activations. Approximate inference of the model would
naturally take the form of a backward pass through the CNN layers, and it
addresses the aforementioned queries in a unified framework.
