  In this paper, we revisit endless online level generation with the recently
proposed experience-driven procedural content generation via reinforcement
learning (EDRL) framework. Inspired by an observation that EDRL tends to
generate recurrent patterns, we formulate a notion of state space closure which
makes any stochastic state appeared possibly in an infinite-horizon online
generation process can be found within a finite-horizon. Through theoretical
analysis, we find that even though state space closure arises a concern about
diversity, it generalises EDRL trained with a finite-horizon to the
infinite-horizon scenario without deterioration of content quality. Moreover,
we verify the quality and the diversity of contents generated by EDRL via
empirical studies, on the widely used Super Mario Bros. benchmark. Experimental
results reveal that the diversity of levels generated by EDRL is limited due to
the state space closure, whereas their quality does not deteriorate in a
horizon which is longer than the one specified in the training. Concluding our
outcomes and analysis, future work on endless online level generation via
reinforcement learning should address the issue of diversity while assuring the
occurrence of state space closure and quality.
