  In reinforcement learning, the optimism in the face of uncertainty (OFU) is a
mainstream principle for directing exploration towards less explored areas,
characterized by higher uncertainty. However, in the presence of environmental
stochasticity (noise), purely optimistic exploration may lead to excessive
probing of high-noise areas, consequently impeding exploration efficiency.
Hence, in exploring noisy environments, while optimism-driven exploration
serves as a foundation, prudent attention to alleviating unnecessary
over-exploration in high-noise areas becomes beneficial. In this work, we
propose Optimistic Value Distribution Explorer (OVD-Explorer) to achieve a
noise-aware optimistic exploration for continuous control. OVD-Explorer
proposes a new measurement of the policy's exploration ability considering
noise in optimistic perspectives, and leverages gradient ascent to drive
exploration. Practically, OVD-Explorer can be easily integrated with continuous
control RL algorithms. Extensive evaluations on the MuJoCo and GridChaos tasks
demonstrate the superiority of OVD-Explorer in achieving noise-aware optimistic
exploration.
