  Meta-gradient Reinforcement Learning (RL) allows agents to self-tune their
hyper-parameters in an online fashion during training. In this paper, we
identify a bias in the meta-gradient of current meta-gradient RL approaches.
This bias comes from using the critic that is trained using the meta-learned
discount factor for the advantage estimation in the outer objective which
requires a different discount factor. Because the meta-learned discount factor
is typically lower than the one used in the outer objective, the resulting bias
can cause the meta-gradient to favor myopic policies. We propose a simple
solution to this issue: we eliminate this bias by using an alternative,
\emph{outer} value function in the estimation of the outer loss. To obtain this
outer value function we add a second head to the critic network and train it
alongside the classic critic, using the outer loss discount factor. On an
illustrative toy problem, we show that the bias can cause catastrophic failure
of current meta-gradient RL approaches, and show that our proposed solution
fixes it. We then apply our method to a more complex environment and
demonstrate that fixing the meta-gradient bias can significantly improve
performance.
