  We introduce a novel training procedure for policy gradient methods wherein
episodic memory is used to optimize the hyperparameters of reinforcement
learning algorithms on-the-fly. Unlike other hyperparameter searches, we
formulate hyperparameter scheduling as a standard Markov Decision Process and
use episodic memory to store the outcome of used hyperparameters and their
training contexts. At any policy update step, the policy learner refers to the
stored experiences, and adaptively reconfigures its learning algorithm with the
new hyperparameters determined by the memory. This mechanism, dubbed as
Episodic Policy Gradient Training (EPGT), enables an episodic learning process,
and jointly learns the policy and the learning algorithm's hyperparameters
within a single run. Experimental results on both continuous and discrete
environments demonstrate the advantage of using the proposed method in boosting
the performance of various policy gradient algorithms.
