  Large-scale pre-training models have promoted the development of
histopathology image analysis. However, existing self-supervised methods for
histopathology images focus on learning patch features, while there is still a
lack of available pre-training models for WSI-level feature learning. In this
paper, we propose a novel self-supervised learning framework for pan-cancer
WSI-level representation pre-training with the designed position-aware masked
autoencoder (PAMA). Meanwhile, we propose the position-aware cross-attention
(PACA) module with a kernel reorientation (KRO) strategy and an anchor dropout
(AD) mechanism. The KRO strategy can capture the complete semantic structure
and eliminate ambiguity in WSIs, and the AD contributes to enhancing the
robustness and generalization of the model. We evaluated our method on 6
large-scale datasets from multiple organs for pan-cancer classification tasks.
The results have demonstrated the effectiveness of PAMA in generalized and
discriminative WSI representation learning and pan-cancer WSI pre-training. The
proposed method was also compared with 7 WSI analysis methods. The experimental
results have indicated that our proposed PAMA is superior to the
state-of-the-art methods.The code and checkpoints are available at
https://github.com/WkEEn/PAMA.
