  This paper addresses the problem of optimizing charging/discharging schedules
of electric vehicles (EVs) when participate in demand response (DR). As there
exist uncertainties in EVs' remaining energy, arrival and departure time, and
future electricity prices, it is quite difficult to make charging decisions to
minimize charging cost while guarantee that the EV's battery
state-of-the-charge (SOC) is within certain range. To handle with this dilemma,
this paper formulates the EV charging scheduling problem as a constrained
Markov decision process (CMDP). By synergistically combining the augmented
Lagrangian method and soft actor critic algorithm, a novel safe off-policy
reinforcement learning (RL) approach is proposed in this paper to solve the
CMDP. The actor network is updated in a policy gradient manner with the
Lagrangian value function. A double-critics network is adopted to synchronously
estimate the action-value function to avoid overestimation bias. The proposed
algorithm does not require strong convexity guarantee of examined problems and
is sample efficient. Comprehensive numerical experiments with real-world
electricity price demonstrate that our proposed algorithm can achieve high
solution optimality and constraints compliance.
