  We propose a distributed algorithm for multiagent systems that aim to
optimize a common objective when agents differ in their estimates of the
objective-relevant state of the environment. Each agent keeps an estimate of
the environment and a model of the behavior of other agents. The model of other
agents' behavior assumes agents choose their actions randomly based on a
stationary distribution determined by the empirical frequencies of past
actions. At each step, each agent takes the action that maximizes its
expectation of the common objective computed with respect to its estimate of
the environment and its model of others. We propose a weighted averaging rule
with non-doubly stochastic weights for agents to estimate the empirical
frequency of past actions of all other agents by exchanging their estimates
with their neighbors over a time-varying communication network. Under this
averaging rule, we show agents' estimates converge to the actual empirical
frequencies fast enough. This implies convergence of actions to a Nash
equilibrium of the game with identical payoffs given by the expectation of the
common objective with respect to an asymptotically agreed estimate of the state
of the environment.
