  Multi-agent reinforcement learning (MARL), despite its popularity and
empirical success, suffers from the curse of dimensionality. This paper builds
the mathematical framework to approximate cooperative MARL by a mean-field
control (MFC) approach, and shows that the approximation error is of
$\mathcal{O}(\frac{1}{\sqrt{N}})$. By establishing an appropriate form of the
dynamic programming principle for both the value function and the Q function,
it proposes a model-free kernel-based Q-learning algorithm (MFC-K-Q), which is
shown to have a linear convergence rate for the MFC problem, the first of its
kind in the MARL literature. It further establishes that the convergence rate
and the sample complexity of MFC-K-Q are independent of the number of agents
$N$, which provides an $\mathcal{O}(\frac{1}{\sqrt{N}})$ approximation to the
MARL problem with $N$ agents in the learning environment. Empirical studies for
the network traffic congestion problem demonstrate that MFC-K-Q outperforms
existing MARL algorithms when $N$ is large, for instance when $N>50$.
