  Masked Autoencoders (MAEs) learn generalizable representations for image,
text, audio, video, etc., by reconstructing masked input data from tokens of
the visible data. Current MAE approaches for videos rely on random patch, tube,
or frame-based masking strategies to select these tokens. This paper proposes
AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our
adaptive masking strategy samples visible tokens based on the semantic context
using an auxiliary sampling network. This network estimates a categorical
distribution over spacetime-patch tokens. The tokens that increase the expected
reconstruction error are rewarded and selected as visible tokens, motivated by
the policy gradient algorithm in reinforcement learning. We show that AdaMAE
samples more tokens from the high spatiotemporal information regions, thereby
allowing us to mask 95% of tokens, resulting in lower memory requirements and
faster pre-training. We conduct ablation studies on the Something-Something v2
(SSv2) dataset to demonstrate the efficacy of our adaptive sampling approach
and report state-of-the-art results of 70.0% and 81.7% in top-1 accuracy on
SSv2 and Kinetics-400 action classification datasets with a ViT-Base backbone
and 800 pre-training epochs.
