  Training agents via off-policy deep reinforcement learning (RL) requires a
large memory, named replay memory, that stores past experiences used for
learning. These experiences are sampled, uniformly or non-uniformly, to create
the batches used for training. When calculating the loss function, off-policy
algorithms assume that all samples are of the same importance. In this paper,
we hypothesize that training can be enhanced by assigning different importance
for each experience based on their temporal-difference (TD) error directly in
the training objective. We propose a novel method that introduces a weighting
factor for each experience when calculating the loss function at the learning
stage. In addition to improving convergence speed when used with uniform
sampling, the method can be combined with prioritization methods for
non-uniform sampling. Combining the proposed method with prioritization methods
improves sampling efficiency while increasing the performance of TD-based
off-policy RL algorithms. The effectiveness of the proposed method is
demonstrated by experiments in six environments of the OpenAI Gym suite. The
experimental results demonstrate that the proposed method achieves a 33%~76%
reduction of convergence speed in three environments and an 11% increase in
returns and a 3%~10% increase in success rate for other three environments.
