  Safe reinforcement learning (RL) with hard constraint guarantees is a
promising optimal control direction for multi-energy management systems. It
only requires the environment-specific constraint functions itself a priori and
not a complete model. The project-specific upfront and ongoing engineering
efforts are therefore still reduced, better representations of the underlying
system dynamics can still be learnt, and modelling bias is kept to a minimum.
However, even the constraint functions alone are not always trivial to
accurately provide in advance, leading to potentially unsafe behaviour. In this
paper, we present two novel advancements: (I) combining the OptLayer and
SafeFallback method, named OptLayerPolicy, to increase the initial utility
while keeping a high sample efficiency and the possibility to formulate
equality constraints. (II) introducing self-improving hard constraints, to
increase the accuracy of the constraint functions as more and new data becomes
available so that better policies can be learnt. Both advancements keep the
constraint formulation decoupled from the RL formulation, so new (presumably
better) RL algorithms can act as drop-in replacements. We have shown that, in a
simulated multi-energy system case study, the initial utility is increased to
92.4% (OptLayerPolicy) compared to 86.1% (OptLayer) and that the policy after
training is increased to 104.9% (GreyOptLayerPolicy) compared to 103.4%
(OptLayer) - all relative to a vanilla RL benchmark. Although introducing
surrogate functions into the optimisation problem requires special attention,
we conclude that the newly presented GreyOptLayerPolicy method is the most
advantageous.
