  Crohn's Disease (CD) and Ulcerative Colitis (UC) are the two main
Inflammatory Bowel Disease (IBD) types. We developed deep learning models to
identify histological disease features for both CD and UC using only endoscopic
labels. We explored fine-tuning and end-to-end training of two state-of-the-art
self-supervised models for predicting three different endoscopic categories (i)
CD vs UC (AUC=0.87), (ii) normal vs lesional (AUC=0.81), (iii) low vs high
disease severity score (AUC=0.80). We produced visual attention maps to
interpret what the models learned and validated them with the support of a
pathologist, where we observed a strong association between the models'
predictions and histopathological inflammatory features of the disease.
Additionally, we identified several cases where the model incorrectly predicted
normal samples as lesional but were correct on the microscopic level when
reviewed by the pathologist. This tendency of histological presentation to be
more severe than endoscopic presentation was previously published in the
literature. In parallel, we utilised a model trained on the Colon Nuclei
Identification and Counting (CoNIC) dataset to predict and explore 6 cell
populations. We observed correlation between areas enriched with the predicted
immune cells in biopsies and the pathologist's feedback on the attention maps.
Finally, we identified several cell level features indicative of disease
severity in CD and UC. These models can enhance our understanding about the
pathology behind IBD and can shape our strategies for patient stratification in
clinical trials.
