  A common problem in machine learning is determining if a variable
significantly contributes to a model's prediction performance. This problem is
aggravated for datasets, such as gene expression datasets, that suffer the
worst case of dimensionality: a low number of observations along with a high
number of possible explanatory variables. In such scenarios, traditional
methods for testing variable statistical significance or constructing variable
confidence intervals do not apply. To address these problems, we developed a
novel permutation framework for testing the significance of variables in
supervised models. Our permutation framework has three main advantages. First,
it is non-parametric and does not rely on distributional assumptions or
asymptotic results. Second, it not only ranks model variables in terms of
relative importance, but also tests for statistical significance of each
variable. Third, it can test for the significance of the interaction between
model variables. We applied this permutation framework to multi-class
classification of the Iris flower dataset and of brain regions in RNA
expression data, and using this framework showed variable-level statistical
significance and interactions.
