  Recent approaches have utilized self-supervised auxiliary tasks as
representation learning to improve the performance and sample efficiency of
vision-based reinforcement learning algorithms in single-agent settings.
However, in multi-agent reinforcement learning (MARL), these techniques face
challenges because each agent only receives partial observation from an
environment influenced by others, resulting in correlated observations in the
agent dimension. So it is necessary to consider agent-level information in
representation learning for MARL. In this paper, we propose an effective
framework called \textbf{M}ulti-\textbf{A}gent \textbf{M}asked
\textbf{A}ttentive \textbf{C}ontrastive \textbf{L}earning (MA2CL), which
encourages learning representation to be both temporal and agent-level
predictive by reconstructing the masked agent observation in latent space.
Specifically, we use an attention reconstruction model for recovering and the
model is trained via contrastive learning. MA2CL allows better utilization of
contextual information at the agent level, facilitating the training of MARL
agents for cooperation tasks. Extensive experiments demonstrate that our method
significantly improves the performance and sample efficiency of different MARL
algorithms and outperforms other methods in various vision-based and
state-based scenarios. Our code can be found in
\url{https://github.com/ustchlsong/MA2CL}
