  Extrinsic rewards can effectively guide reinforcement learning (RL) agents in
specific tasks. However, extrinsic rewards frequently fall short in complex
environments due to the significant human effort needed for their design and
annotation. This limitation underscores the necessity for intrinsic rewards,
which offer auxiliary and dense signals and can enable agents to learn in an
unsupervised manner. Although various intrinsic reward formulations have been
proposed, their implementation and optimization details are insufficiently
explored and lack standardization, thereby hindering research progress. To
address this gap, we introduce RLeXplore, a unified, highly modularized, and
plug-and-play framework offering reliable implementations of eight
state-of-the-art intrinsic reward algorithms. Furthermore, we conduct an
in-depth study that identifies critical implementation details and establishes
well-justified standard practices in intrinsically-motivated RL. The source
code for RLeXplore is available at https://github.com/RLE-Foundation/RLeXplore.
