  In typical reinforcement learning (RL), the environment is assumed given and
the goal of the learning is to identify an optimal policy for the agent taking
actions through its interactions with the environment. In this paper, we extend
this setting by considering the environment is not given, but controllable and
learnable through its interaction with the agent at the same time. This
extension is motivated by environment design scenarios in the real-world,
including game design, shopping space design and traffic signal design.
Theoretically, we find a dual Markov decision process (MDP) w.r.t. the
environment to that w.r.t. the agent, and derive a policy gradient solution to
optimizing the parametrized environment. Furthermore, discontinuous
environments are addressed by a proposed general generative framework. Our
experiments on a Maze game design task show the effectiveness of the proposed
algorithms in generating diverse and challenging Mazes against various agent
settings.
