  The reinforcement learning (RL) problem is rife with sources of
non-stationarity, making it a notoriously difficult problem domain for the
application of neural networks. We identify a mechanism by which non-stationary
prediction targets can prevent learning progress in deep RL agents:
\textit{capacity loss}, whereby networks trained on a sequence of target values
lose their ability to quickly update their predictions over time. We
demonstrate that capacity loss occurs in a range of RL agents and environments,
and is particularly damaging to performance in sparse-reward tasks. We then
present a simple regularizer, Initial Feature Regularization (InFeR), that
mitigates this phenomenon by regressing a subspace of features towards its
value at initialization, leading to significant performance improvements in
sparse-reward environments such as Montezuma's Revenge. We conclude that
preventing capacity loss is crucial to enable agents to maximally benefit from
the learning signals they obtain throughout the entire training trajectory.
