  For reinforcement learning (RL), it is challenging for an agent to master a
task that requires a specific series of actions due to sparse rewards. To solve
this problem, reverse curriculum generation (RCG) provides a reverse expansion
approach that automatically generates a curriculum for the agent to learn. More
specifically, RCG adapts the initial state distribution from the neighborhood
of a goal to a distance as training proceeds. However, the initial state
distribution generated for each iteration might be biased, thus making the
policy overfit or slowing down the reverse expansion rate. While training RCG
for actor-critic (AC) based RL algorithms, this poor generalization and slow
convergence might be induced by the tight coupling between an AC pair.
Therefore, we propose a parallelized approach that simultaneously trains
multiple AC pairs and periodically exchanges their critics. We empirically
demonstrate that this proposed approach can improve RCG in performance and
convergence, and it can also be applied to other AC based RL algorithms with
adapted initial state distribution.
