  Though neural network models demonstrate impressive performance, we do not
understand exactly how these black-box models make individual predictions. This
drawback has led to substantial research devoted to understand these models in
areas such as robustness, interpretability, and generalization ability. In this
paper, we consider the problem of exploring the prediction level sets of a
classifier using probabilistic programming. We define a prediction level set to
be the set of examples for which the predictor has the same specified
prediction confidence with respect to some arbitrary data distribution.
Notably, our sampling-based method does not require the classifier to be
differentiable, making it compatible with arbitrary classifiers. As a specific
instantiation, if we take the classifier to be a neural network and the data
distribution to be that of the training data, we can obtain examples that will
result in specified predictions by the neural network. We demonstrate this
technique with experiments on a synthetic dataset and MNIST. Such level sets in
classification may facilitate human understanding of classification behaviors.
