  We consider a finite number of $N$ statistically equal agents, each moving on
a finite set of states according to a continuous-time Markov Decision Process
(MDP). Transition intensities of the agents and generated rewards depend not
only on the state and action of the agent itself, but also on the states of the
other agents as well as the chosen action. Interactions like this are typical
for a wide range of models in e.g. biology, epidemics, finance, social science
and queueing systems among others. The aim is to maximize the expected
discounted reward of the system, i.e. the agents have to cooperate as a team.
Computationally this is a difficult task when $N$ is large. Thus, we consider
the limit for $N\to\infty.$ In contrast to other papers we treat this problem
from an MDP perspective. This has the advantage that we need less regularity
assumptions in order to construct asymptotically optimal strategies than using
viscosity solutions of HJB equations. The convergence rate is $1/\sqrt{N}$. We
show how to apply our results using two examples: a machine replacement problem
and a problem from epidemics. We also show that optimal feedback policies from
the limiting problem are not necessarily asymptotically optimal.
