  This study evaluates the effect of counterfactual explanations on the
interpretation of chest X-rays. We conduct a reader study with two radiologists
assessing 240 chest X-ray predictions to rate their confidence that the model's
prediction is correct using a 5 point scale. Half of the predictions are false
positives. Each prediction is explained twice, once using traditional
attribution methods and once with a counterfactual explanation. The overall
results indicate that counterfactual explanations allow a radiologist to have
more confidence in true positive predictions compared to traditional approaches
(0.15$\pm$0.95 with p=0.01) with only a small increase in false positive
predictions (0.04$\pm$1.06 with p=0.57). We observe the specific prediction
tasks of Mass and Atelectasis appear to benefit the most compared to other
tasks.
