  Processing giga-pixel whole slide histopathology images (WSI) is a
computationally expensive task. Multiple instance learning (MIL) has become the
conventional approach to process WSIs, in which these images are split into
smaller patches for further processing. However, MIL-based techniques ignore
explicit information about the individual cells within a patch. In this paper,
by defining the novel concept of shared-context processing, we designed a
multi-modal Graph Transformer (AMIGO) that uses the celluar graph within the
tissue to provide a single representation for a patient while taking advantage
of the hierarchical structure of the tissue, enabling a dynamic focus between
cell-level and tissue-level information. We benchmarked the performance of our
model against multiple state-of-the-art methods in survival prediction and
showed that ours can significantly outperform all of them including
hierarchical Vision Transformer (ViT). More importantly, we show that our model
is strongly robust to missing information to an extent that it can achieve the
same performance with as low as 20% of the data. Finally, in two different
cancer datasets, we demonstrated that our model was able to stratify the
patients into low-risk and high-risk groups while other state-of-the-art
methods failed to achieve this goal. We also publish a large dataset of
immunohistochemistry images (InUIT) containing 1,600 tissue microarray (TMA)
cores from 188 patients along with their survival information, making it one of
the largest publicly available datasets in this context.
