  Inflammatory bowel disease (IBD), in particular ulcerative colitis (UC), is
graded by endoscopists and this assessment is the basis for risk stratification
and therapy monitoring. Presently, endoscopic characterisation is largely
operator dependant leading to sometimes undesirable clinical outcomes for
patients with IBD. We focus on the Mayo Endoscopic Scoring (MES) system which
is widely used but requires the reliable identification of subtle changes in
mucosal inflammation. Most existing deep learning classification methods cannot
detect these fine-grained changes which make UC grading such a challenging
task. In this work, we introduce a novel patch-level instance-group
discrimination with pretext-invariant representation learning (PLD-PIRL) for
self-supervised learning (SSL). Our experiments demonstrate both improved
accuracy and robustness compared to the baseline supervised network and several
state-of-the-art SSL methods. Compared to the baseline (ResNet50) supervised
classification our proposed PLD-PIRL obtained an improvement of 4.75% on
hold-out test data and 6.64% on unseen center test data for top-1 accuracy.
