  Deriving interpretable prognostic features from deep-learning-based
prognostic histopathology models remains a challenge. In this study, we
developed a deep learning system (DLS) for predicting disease specific survival
for stage II and III colorectal cancer using 3,652 cases (27,300 slides). When
evaluated on two validation datasets containing 1,239 cases (9,340 slides) and
738 cases (7,140 slides) respectively, the DLS achieved a 5-year
disease-specific survival AUC of 0.70 (95%CI 0.66-0.73) and 0.69 (95%CI
0.64-0.72), and added significant predictive value to a set of 9
clinicopathologic features. To interpret the DLS, we explored the ability of
different human-interpretable features to explain the variance in DLS scores.
We observed that clinicopathologic features such as T-category, N-category, and
grade explained a small fraction of the variance in DLS scores (R2=18% in both
validation sets). Next, we generated human-interpretable histologic features by
clustering embeddings from a deep-learning based image-similarity model and
showed that they explain the majority of the variance (R2 of 73% to 80%).
Furthermore, the clustering-derived feature most strongly associated with high
DLS scores was also highly prognostic in isolation. With a distinct visual
appearance (poorly differentiated tumor cell clusters adjacent to adipose
tissue), this feature was identified by annotators with 87.0-95.5% accuracy.
Our approach can be used to explain predictions from a prognostic deep learning
model and uncover potentially-novel prognostic features that can be reliably
identified by people for future validation studies.
