  We propose the conditional predictive impact (CPI), a consistent and unbiased
estimator of the association between one or several features and a given
outcome, conditional on a reduced feature set. Building on the knockoff
framework of Cand\`es et al. (2018), we develop a novel testing procedure that
works in conjunction with any valid knockoff sampler, supervised learning
algorithm, and loss function. The CPI can be efficiently computed for
high-dimensional data without any sparsity constraints. We demonstrate
convergence criteria for the CPI and develop statistical inference procedures
for evaluating its magnitude, significance, and precision. These tests aid in
feature and model selection, extending traditional frequentist and Bayesian
techniques to general supervised learning tasks. The CPI may also be applied in
causal discovery to identify underlying multivariate graph structures. We test
our method using various algorithms, including linear regression, neural
networks, random forests, and support vector machines. Empirical results show
that the CPI compares favorably to alternative variable importance measures and
other nonparametric tests of conditional independence on a diverse array of
real and simulated datasets. Simulations confirm that our inference procedures
successfully control Type I error and achieve nominal coverage probability. Our
method has been implemented in an R package, cpi, which can be downloaded from
https://github.com/dswatson/cpi.
