  Reward shaping addresses the challenge of sparse rewards in reinforcement
learning by constructing denser and more informative reward signals. To achieve
self-adaptive and highly efficient reward shaping, we propose a novel method
that incorporates success rates derived from historical experiences into shaped
rewards. Our approach utilizes success rates sampled from Beta distributions,
which dynamically evolve from uncertain to reliable values as more data is
collected. Initially, the self-adaptive success rates exhibit more randomness
to encourage exploration. Over time, they become more certain to enhance
exploitation, thus achieving a better balance between exploration and
exploitation. We employ Kernel Density Estimation (KDE) combined with Random
Fourier Features (RFF) to derive the Beta distributions, resulting in a
computationally efficient implementation in high-dimensional continuous state
spaces. This method provides a non-parametric and learning-free approach. The
proposed method is evaluated on a wide range of continuous control tasks with
sparse and delayed rewards, demonstrating significant improvements in sample
efficiency and convergence stability compared to relevant baselines.
