  This paper studies stochastic games on large graphs and their graphon limits.
We propose a new formulation of graphon games based on a single typical
player's label-state distribution. In contrast, other recently proposed models
of graphon games work directly with a continuum of players, which involves
serious measure-theoretic technicalities. In fact, by viewing the label as a
component of the state process, we show in our formulation that graphon games
are a special case of mean field games, albeit with certain inevitable
degeneracies and discontinuities that make most existing results on mean field
games inapplicable. Nonetheless, we prove existence of Markovian graphon
equilibria under fairly general assumptions, as well as uniqueness under a
monotonicity condition. Most imporantly, we show how our notion of graphon
equilibrium can be used to construct approximate equilibria for large finite
games set on any (weighted, directed) graph which converges in cut norm. The
lack of players' exchangeability necessitates a careful definition of
approximate equilibrium, allowing heterogeneity among the players'
approximation errors, and we show how various regularity properties of the
model inputs and underlying graphon lead naturally to different strengths of
approximation.
