  We propose a discrete time graphon game formulation on continuous state and
action spaces using a representative player to study stochastic games with
heterogeneous interaction among agents. This formulation admits both
philosophical and mathematical advantages, compared to a widely adopted
formulation using a continuum of players. We prove the existence and uniqueness
of the graphon equilibrium with mild assumptions, and show that this
equilibrium can be used to construct an approximate solution for finite player
game on networks, which is challenging to analyze and solve due to curse of
dimensionality. An online oracle-free learning algorithm is developed to solve
the equilibrium numerically, and sample complexity analysis is provided for its
convergence.
