  While reinforcement learning (RL) is gaining popularity in energy systems
control, its real-world applications are limited due to the fact that the
actions from learned policies may not satisfy functional requirements or be
feasible for the underlying physical system. In this work, we propose PROjected
Feasibility (PROF), a method to enforce convex operational constraints within
neural policies. Specifically, we incorporate a differentiable projection layer
within a neural network-based policy to enforce that all learned actions are
feasible. We then update the policy end-to-end by propagating gradients through
this differentiable projection layer, making the policy cognizant of the
operational constraints. We demonstrate our method on two applications:
energy-efficient building operation and inverter control. In the building
operation setting, we show that PROF maintains thermal comfort requirements
while improving energy efficiency by 4% over state-of-the-art methods. In the
inverter control setting, PROF perfectly satisfies voltage constraints on the
IEEE 37-bus feeder system, as it learns to curtail as little renewable energy
as possible within its safety set.
