  The interactions between tumor cells and the tumor microenvironment (TME)
dictate therapeutic efficacy of radiation and many systemic therapies in breast
cancer. However, to date, there is not a widely available method to
reproducibly measure tumor and immune phenotypes for each patient's tumor.
Given this unmet clinical need, we applied multiple instance learning (MIL)
algorithms to assess activity of ten biologically relevant pathways from the
hematoxylin and eosin (H&E) slide of primary breast tumors. We employed
different feature extraction approaches and state-of-the-art model
architectures. Using binary classification, our models attained area under the
receiver operating characteristic (AUROC) scores above 0.70 for nearly all gene
expression pathways and on some cases, exceeded 0.80. Attention maps suggest
that our trained models recognize biologically relevant spatial patterns of
cell sub-populations from H&E. These efforts represent a first step towards
developing computational H&E biomarkers that reflect facets of the TME and hold
promise for augmenting precision oncology.
