  Machine learning practice is often impacted by confounders. Confounding can
be particularly severe in remote digital health studies where the participants
self-select to enter the study. While many different confounding adjustment
approaches have been proposed in the literature, most of these methods rely on
modeling assumptions, and it is unclear how robust they are to violations of
these assumptions. This realization has recently motivated the development of
restricted permutation methods to quantify the influence of observed
confounders on the predictive performance of a machine learning models and
evaluate if confounding adjustment methods are working as expected. In this
paper we show, nonetheless, that restricted permutations can generate biased
estimates of the contribution of the confounders to the predictive performance
of a learner, and we propose an alternative approach to tackle this problem. By
viewing a classification task from a causality perspective, we are able to
leverage conditional independence tests between predictions and test set labels
and confounders in order to detect confounding on the predictive performance of
a classifier. We illustrate the application of our causality-based approach to
data collected from mHealth study in Parkinson's disease.
