  Causal influence measures for machine learnt classifiers shed light on the
reasons behind classification, and aid in identifying influential input
features and revealing their biases. However, such analyses involve evaluating
the classifier using datapoints that may be atypical of its training
distribution. Standard methods for training classifiers that minimize empirical
risk do not constrain the behavior of the classifier on such datapoints. As a
result, training to minimize empirical risk does not distinguish among
classifiers that agree on predictions in the training distribution but have
wildly different causal influences. We term this problem covariate shift in
causal testing and formally characterize conditions under which it arises. As a
solution to this problem, we propose a novel active learning algorithm that
constrains the influence measures of the trained model. We prove that any two
predictors whose errors are close on both the original training distribution
and the distribution of atypical points are guaranteed to have causal
influences that are also close. Further, we empirically demonstrate with
synthetic labelers that our algorithm trains models that (i) have similar
causal influences as the labeler's model, and (ii) generalize better to
out-of-distribution points while (iii) retaining their accuracy on
in-distribution points.
