  We consider the problem of maximizing the expected average reward obtained
over an infinite time horizon by $n$ weakly coupled Markov decision processes.
Our setup is a substantial generalization of the multi-armed restless bandit
problem that allows for multiple actions and constraints. We establish a
connection with a deterministic and continuous-variable control problem where
the objective is to maximize the average reward derived from an occupancy
measure that represents the empirical distribution of the processes when $n \to
\infty$. We show that a solution of this fluid problem can be used to construct
policies for the weakly coupled processes that achieve the maximum expected
average reward as $n \to \infty$, and we give sufficient conditions for the
existence of solutions. Under certain assumptions on the constraints, we prove
that these conditions are automatically satisfied if the unconstrained
single-process problem admits a suitable unichain and aperiodic policy. In
particular, the assumptions include multi-armed restless bandits and a broad
class of problems with multiple actions and inequality constraints. Also, the
policies can be constructed in an explicit way in these cases. Our theoretical
results are complemented by several concrete examples and numerical
experiments, which include multichain setups that are covered by the
theoretical results.
