  Accurate identification of synergistic treatment combinations and their
underlying biological mechanisms is critical across many disease domains,
especially cancer. In translational oncology research, preclinical systems such
as patient-derived xenografts (PDX) have emerged as a unique study design
evaluating multiple treatments administered to samples from the same human
tumor implanted into genetically identical mice. In this paper, we propose a
novel Bayesian probabilistic tree-based framework for PDX data to investigate
the hierarchical relationships between treatments by inferring treatment
cluster trees, referred to as treatment trees (Rx-tree). The framework
motivates a new metric of mechanistic similarity between two or more treatments
accounting for inherent uncertainty in tree estimation; treatments with a high
estimated similarity have potentially high mechanistic synergy. Building upon
Dirichlet Diffusion Trees, we derive a closed-form marginal likelihood encoding
the tree structure, which facilitates computationally efficient posterior
inference via a new two-stage algorithm. Simulation studies demonstrate
superior performance of the proposed method in recovering the tree structure
and treatment similarities. Our analyses of a recently collated PDX dataset
produce treatment similarity estimates that show a high degree of concordance
with known biological mechanisms across treatments in five different cancers.
More importantly, we uncover new and potentially effective combination
therapies that confer synergistic regulation of specific downstream biological
pathways for future clinical investigations. Our accompanying code, data, and
shiny application for visualization of results are available at:
https://github.com/bayesrx/RxTree.
