  This paper proposes a deep learning-based optimal battery management scheme
for frequency regulation (FR) by integrating model predictive control (MPC),
supervised learning (SL), reinforcement learning (RL), and high-fidelity
battery models. By taking advantage of deep neural networks (DNNs), the derived
DNN-approximated policy is computationally efficient in online implementation.
The design procedure of the proposed scheme consists of two sequential
processes: (1) the SL process, in which we first run a simulation with an MPC
embedding a low-fidelity battery model to generate a training data set, and
then, based on the generated data set, we optimize a DNN-approximated policy
using SL algorithms; and (2) the RL process, in which we utilize RL algorithms
to improve the performance of the DNN-approximated policy by balancing
short-term economic incentives and long-term battery degradation. The SL
process speeds up the subsequent RL process by providing a good initialization.
By utilizing RL algorithms, one prominent property of the proposed scheme is
that it can learn from the data generated by simulating the FR policy on the
high-fidelity battery simulator to adjust the DNN-approximated policy, which is
originally based on low-fidelity battery model. A case study using real-world
data of FR signals and prices is performed. Simulation results show that,
compared to conventional MPC schemes, the proposed deep learning-based scheme
can effectively achieve higher economic benefits of FR participation while
maintaining lower online computational cost.
