  Valid statistical inference is crucial for decision-making but difficult to
obtain in supervised learning with multimodal data, e.g., combinations of
clinical features, genomic data, and medical images. Multimodal data often
warrants the use of black-box algorithms, for instance, random forests or
neural networks, which impede the use of traditional variable significance
tests. We address this problem by proposing the use of COvariance Measure Tests
(COMETs), which are calibrated and powerful tests that can be combined with any
sufficiently predictive supervised learning algorithm. We apply COMETs to
several high-dimensional, multimodal data sets to illustrate (i) variable
significance testing for finding relevant mutations modulating drug-activity,
(ii) modality selection for predicting survival in liver cancer patients with
multiomics data, and (iii) modality selection with clinical features and
medical imaging data. In all applications, COMETs yield results consistent with
domain knowledge without requiring data-driven pre-processing which may
invalidate type I error control. These novel applications with high-dimensional
multimodal data corroborate prior results on the power and robustness of COMETs
for significance testing. COMETs are implemented in the comets R package
available on CRAN and pycomets Python library available on GitHub. Source code
for reproducing all results is available at
https://github.com/LucasKook/comets. All data sets used in this work are openly
available.
