  Studying the human microbiome has gained substantial interest in recent
years, and a common task in the analysis of these data is to cluster microbiome
compositions into subtypes. This subdivision of samples into subgroups serves
as an intermediary step in achieving personalized diagnosis and treatment. In
applying existing clustering methods to modern microbiome studies including the
American Gut Project (AGP) data, we found that this seemingly standard task,
however, is very challenging in the microbiome composition context due to
several key features of such data. Standard distance-based clustering
algorithms generally do not produce reliable results as they do not take into
account the heterogeneity of the cross-sample variability among the bacterial
taxa, while existing model-based approaches do not allow sufficient flexibility
for the identification of complex within-cluster variation from cross-cluster
variation. Direct applications of such methods generally lead to overly
dispersed clusters in the AGP data and such phenomenon is common for other
microbiome data. To overcome these challenges, we introduce Dirichlet-tree
multinomial mixtures (DTMM) as a Bayesian generative model for clustering
amplicon sequencing data in microbiome studies. DTMM models the microbiome
population with a mixture of Dirichlet-tree kernels that utilizes the
phylogenetic tree to offer a more flexible covariance structure in
characterizing within-cluster variation, and it provides a means for
identifying a subset of signature taxa that distinguish the clusters. We
perform extensive simulation studies to evaluate the performance of DTMM and
compare it to state-of-the-art model-based and distance-based clustering
methods in the microbiome context. Finally, we report a case study on the fecal
data from the AGP to identify compositional clusters among individuals with
inflammatory bowel disease and diabetes.
