  Although deep reinforcement learning has demonstrated impressive achievements
in controlling various autonomous systems, e.g., autonomous vehicles or
humanoid robots, its inherent reliance on random exploration raises safety
concerns in their real-world applications. To improve system safety during the
learning process, a variety of Safe Reinforcement Learning (SRL) algorithms
have been proposed, which usually incorporate safety constraints within the
Constrained Markov Decision Process (CMDP) framework. However, the efficacy of
these SRL algorithms often relies on accurate function approximations, a task
that is notably challenging to accomplish in the early learning stages due to
data insufficiency. To address this problem, we introduce a Genralizable Safety
enhancer (GenSafe) in this work. Leveraging model order reduction techniques,
we first construct a Reduced Order Markov Decision Process (ROMDP) as a
low-dimensional proxy for the original cost function in CMDP. Then, by solving
ROMDP-based constraints that are reformulated from the original cost
constraints, the proposed GenSafe refines the actions taken by the agent to
enhance the possibility of constraint satisfaction. Essentially, GenSafe acts
as an additional safety layer for SRL algorithms, offering broad compatibility
across diverse SRL approaches. The performance of GenSafe is examined on
multiple SRL benchmark problems. The results show that, it is not only able to
improve the safety performance, especially in the early learning phases, but
also to maintain the task performance at a satisfactory level.
