  In multi-agent systems, agents need to interact and collaborate with other
agents in environments. Agent modeling is crucial to facilitate agent
interactions and make adaptive cooperation strategies. However, it is
challenging for agents to model the beliefs, behaviors, and intentions of other
agents in non-stationary environment where all agent policies are learned
simultaneously. In addition, the existing methods realize agent modeling
through behavior cloning which assume that the local information of other
agents can be accessed during execution or training. However, this assumption
is infeasible in unknown scenarios characterized by unknown agents, such as
competition teams, unreliable communication and federated learning due to
privacy concerns. To eliminate this assumption and achieve agent modeling in
unknown scenarios, Fact-based Agent modeling (FAM) method is proposed in which
fact-based belief inference (FBI) network models other agents in partially
observable environment only based on its local information. The reward and
observation obtained by agents after taking actions are called facts, and FAM
uses facts as reconstruction target to learn the policy representation of other
agents through a variational autoencoder. We evaluate FAM on various Multiagent
Particle Environment (MPE) and compare the results with several
state-of-the-art MARL algorithms. Experimental results show that compared with
baseline methods, FAM can effectively improve the efficiency of agent policy
learning by making adaptive cooperation strategies in multi-agent reinforcement
learning tasks, while achieving higher returns in complex
competitive-cooperative mixed scenarios.
