  Recently, convolutional neural networks (CNNs) have shown great success on
the task of monocular depth estimation. A fundamental yet unanswered question
is: how CNNs can infer depth from a single image. Toward answering this
question, we consider visualization of inference of a CNN by identifying
relevant pixels of an input image to depth estimation. We formulate it as an
optimization problem of identifying the smallest number of image pixels from
which the CNN can estimate a depth map with the minimum difference from the
estimate from the entire image. To cope with a difficulty with optimization
through a deep CNN, we propose to use another network to predict those relevant
image pixels in a forward computation. In our experiments, we first show the
effectiveness of this approach, and then apply it to different depth estimation
networks on indoor and outdoor scene datasets. The results provide several
findings that help exploration of the above question.
