  Despite the successes and progress of deep reinforcement learning over the
last decade, several challenges remain that hinder its broader application.
Some fundamental aspects to improve include data efficiency, generalization
capability, and ability to learn in sparse-reward environments, which often
require human-designed dense rewards. Meta-learning has emerged as a promising
approach to address these issues by optimizing components of the learning
algorithm to meet desired characteristics. Additionally, a different line of
work has extensively studied the use of intrinsic rewards to enhance the
exploration capabilities of algorithms. This work investigates how
meta-learning can improve the training signal received by RL agents. The focus
is on meta-learning intrinsic rewards under a framework that doesn't rely on
the use of meta-gradients. We analyze and compare this approach to the use of
extrinsic rewards and a meta-learned advantage function. The developed
algorithms are evaluated on distributions of continuous control tasks with both
parametric and non-parametric variations, and with only sparse rewards
accessible for the evaluation tasks.
