  Recent results in Reinforcement Learning (RL) have shown that agents with
limited training environments are susceptible to a large amount of overfitting
across many domains. A key challenge for RL generalization is to quantitatively
explain the effects of changing parameters on testing performance. Such
parameters include architecture, regularization, and RL-dependent variables
such as discount factor and action stochasticity. We provide empirical results
that show complex and interdependent relationships between hyperparameters and
generalization. We further show that several empirical metrics such as gradient
cosine similarity and trajectory-dependent metrics serve to provide intuition
towards these results.
