  Policies produced by deep reinforcement learning are typically characterised
by their learning curves, but they remain poorly understood in many other
respects. ReLU-based policies result in a partitioning of the input space into
piecewise linear regions. We seek to understand how observed region counts and
their densities evolve during deep reinforcement learning using empirical
results that span a range of continuous control tasks and policy network
dimensions. Intuitively, we may expect that during training, the region density
increases in the areas that are frequently visited by the policy, thereby
affording fine-grained control. We use recent theoretical and empirical results
for the linear regions induced by neural networks in supervised learning
settings for grounding and comparison of our results. Empirically, we find that
the region density increases only moderately throughout training, as measured
along fixed trajectories coming from the final policy. However, the
trajectories themselves also increase in length during training, and thus the
region densities decrease as seen from the perspective of the current
trajectory. Our findings suggest that the complexity of deep reinforcement
learning policies does not principally emerge from a significant growth in the
complexity of functions observed on-and-around trajectories of the policy.
