  Traditional multi-agent reinforcement learning algorithms are not scalable to
environments with more than a few agents, since these algorithms are
exponential in the number of agents. Recent research has introduced successful
methods to scale multi-agent reinforcement learning algorithms to many agent
scenarios using mean field theory. Previous work in this field assumes that an
agent has access to exact cumulative metrics regarding the mean field behaviour
of the system, which it can then use to take its actions. In this paper, we
relax this assumption and maintain a distribution to model the uncertainty
regarding the mean field of the system. We consider two different settings for
this problem. In the first setting, only agents in a fixed neighbourhood are
visible, while in the second setting, the visibility of agents is determined at
random based on distances. For each of these settings, we introduce a
Q-learning based algorithm that can learn effectively. We prove that this
Q-learning estimate stays very close to the Nash Q-value (under a common set of
assumptions) for the first setting. We also empirically show our algorithms
outperform multiple baselines in three different games in the MAgents
framework, which supports large environments with many agents learning
simultaneously to achieve possibly distinct goals.
