  Self-supervised learning has attracted increasing attention as it learns
data-driven representation from data without annotations. Vision
transformer-based autoencoder (ViT-AE) by He et al. (2021) is a recent
self-supervised learning technique that employs a patch-masking strategy to
learn a meaningful latent space. In this paper, we focus on improving ViT-AE
(nicknamed ViT-AE++) for a more effective representation of 2D and 3D medical
images. We propose two new loss functions to enhance the representation during
training. The first loss term aims to improve self-reconstruction by
considering the structured dependencies and indirectly improving the
representation. The second loss term leverages contrastive loss to optimize the
representation from two randomly masked views directly. We extended ViT-AE++ to
a 3D fashion for volumetric medical images as an independent contribution. We
extensively evaluate ViT-AE++ on both natural images and medical images,
demonstrating consistent improvement over vanilla ViT-AE and its superiority
over other contrastive learning approaches. Codes are here:
https://github.com/chinmay5/vit_ae_plus_plus.git.
