  The visual pathway of human brain includes two sub-pathways, ie, the ventral
pathway and the dorsal pathway, which focus on object identification and
dynamic information modeling, respectively. Both pathways comprise multi-layer
structures, with each layer responsible for processing different aspects of
visual information. Inspired by visual information processing mechanism of the
human brain, we propose the Brain Inspired Masked Modeling (BIMM) framework,
aiming to learn comprehensive representations from videos. Specifically, our
approach consists of ventral and dorsal branches, which learn image and video
representations, respectively. Both branches employ the Vision Transformer
(ViT) as their backbone and are trained using masked modeling method. To
achieve the goals of different visual cortices in the brain, we segment the
encoder of each branch into three intermediate blocks and reconstruct
progressive prediction targets with light weight decoders. Furthermore, drawing
inspiration from the information-sharing mechanism in the visual pathways, we
propose a partial parameter sharing strategy between the branches during
training. Extensive experiments demonstrate that BIMM achieves superior
performance compared to the state-of-the-art methods.
