  In this paper, we propose a novel model-free reinforcement learning algorithm
to compute the optimal policies for a multi-agent system with $N$ cooperative
agents where each agent privately observes it's own private type and publicly
observes each others' actions. The goal is to maximize their collective reward.
The problem belongs to the broad class of decentralized control problems with
partial information. We use the common agent approach wherein some fictitious
common agent picks the best policy based on a belief on the current states of
the agents. These beliefs are updated individually for each agent from their
current belief and action histories. Belief state updates without the knowledge
of system dynamics is a challenge. In this paper, we employ particle filters
called the bootstrap filter distributively across agents to update the belief.
We provide a model-free reinforcement learning (RL) method for this multi-agent
partially observable Markov decision processes using the particle filter and
sampled trajectories to estimate the optimal policies for the agents. We
showcase our results with the help of a smartgrid application where the users
strive to reduce collective cost of power for all the agents in the grid.
Finally, we compare the performances for model and model-free implementation of
the RL algorithm establishing the effectiveness of particle filter (pf) method.
