  We address the crucial yet underexplored stability properties of the
Hamilton--Jacobi--Bellman (HJB) equation in model-free reinforcement learning
contexts, specifically for Lipschitz continuous optimal control problems. We
bridge the gap between Lipschitz continuous optimal control problems and
classical optimal control problems in the viscosity solutions framework,
offering new insights into the stability of the value function of Lipschitz
continuous optimal control problems. By introducing structural assumptions on
the dynamics and reward functions, we further study the rate of convergence of
value functions. Moreover, we introduce a generalized framework for Lipschitz
continuous control problems that incorporates the original problem and leverage
it to propose a new HJB-based reinforcement learning algorithm. The stability
properties and performance of the proposed method are tested with well-known
benchmark examples in comparison with existing approaches.
