  We present a Reinforcement Learning (RL) algorithm to solve infinite horizon
asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our
approach can be described as a unified two-timescale Mean Field Q-learning: The
\emph{same} algorithm can learn either the MFG or the MFC solution by simply
tuning the ratio of two learning parameters. The algorithm is in discrete time
and space where the agent not only provides an action to the environment but
also a distribution of the state in order to take into account the mean field
feature of the problem. Importantly, we assume that the agent can not observe
the population's distribution and needs to estimate it in a model-free manner.
The asymptotic MFG and MFC problems are also presented in continuous time and
space, and compared with classical (non-asymptotic or stationary) MFG and MFC
problems. They lead to explicit solutions in the linear-quadratic (LQ) case
that are used as benchmarks for the results of our algorithm.
