  In the machine learning culture, an independent test set is required for
proper model verification. Failures in model verification, including test set
omission and test set leakage, make it impossible to know whether or not a
trained model is fit for purpose. In this article, we present a systematic
review and quantitative analysis of human gut microbiome classification
studies, conducted to measure the frequency and impact of test set omission and
test set leakage on area under the receiver operating curve (AUC) reporting.
Among 102 articles included for analysis, we find that only 12% of studies
report a bona fide test set AUC, meaning that the published AUCs for 88% of
studies cannot be trusted at face value. Our findings cast serious doubt on the
general validity of research claiming that the gut microbiome has high
diagnostic or prognostic potential in human disease.
