  We often desire our models to be interpretable as well as accurate. Prior
work on optimizing models for interpretability has relied on easy-to-quantify
proxies for interpretability, such as sparsity or the number of operations
required. In this work, we optimize for interpretability by directly including
humans in the optimization loop. We develop an algorithm that minimizes the
number of user studies to find models that are both predictive and
interpretable and demonstrate our approach on several data sets. Our human
subjects results show trends towards different proxy notions of
interpretability on different datasets, which suggests that different proxies
are preferred on different tasks.
