  Learning good representation of giga-pixel level whole slide pathology images
(WSI) for downstream tasks is critical. Previous studies employ multiple
instance learning (MIL) to represent WSIs as bags of sampled patches because,
for most occasions, only slide-level labels are available, and only a tiny
region of the WSI is disease-positive area. However, WSI representation
learning still remains an open problem due to: (1) patch sampling on a higher
resolution may be incapable of depicting microenvironment information such as
the relative position between the tumor cells and surrounding tissues, while
patches at lower resolution lose the fine-grained detail; (2) extracting
patches from giant WSI results in large bag size, which tremendously increases
the computational cost. To solve the problems, this paper proposes a
hierarchical-based multimodal transformer framework that learns a hierarchical
mapping between pathology images and corresponding genes. Precisely, we
randomly extract instant-level patch features from WSIs with different
magnification. Then a co-attention mapping between imaging and genomics is
learned to uncover the pairwise interaction and reduce the space complexity of
imaging features. Such early fusion makes it computationally feasible to use
MIL Transformer for the survival prediction task. Our architecture requires
fewer GPU resources compared with benchmark methods while maintaining better
WSI representation ability. We evaluate our approach on five cancer types from
the Cancer Genome Atlas database and achieved an average c-index of $0.673$,
outperforming the state-of-the-art multimodality methods.
