  In practical applications of machine learning, it is necessary to look beyond
standard metrics such as test accuracy in order to validate various qualitative
properties of a model. Partial dependence plots (PDP), including
instance-specific PDPs (i.e., ICE plots), have been widely used as a visual
tool to understand or validate a model. Yet, current PDPs suffer from two main
drawbacks: (1) a user must manually sort or select interesting plots, and (2)
PDPs are usually limited to plots along a single feature. To address these
drawbacks, we formalize a method for automating the selection of interesting
PDPs and extend PDPs beyond showing single features to show the model response
along arbitrary directions, for example in raw feature space or a latent space
arising from some generative model. We demonstrate the usefulness of our
automated dependence plots (ADP) across multiple use-cases and datasets
including model selection, bias detection, understanding out-of-sample
behavior, and exploring the latent space of a generative model.
