  Driven by the global decarbonization effort, the rapid integration of
renewable energy into the conventional electricity grid presents new challenges
and opportunities for the battery energy storage system (BESS) participating in
the energy market. Energy arbitrage can be a significant source of revenue for
the BESS due to the increasing price volatility in the spot market caused by
the mismatch between renewable generation and electricity demand. In addition,
the Frequency Control Ancillary Services (FCAS) markets established to
stabilize the grid can offer higher returns for the BESS due to their
capability to respond within milliseconds. Therefore, it is crucial for the
BESS to carefully decide how much capacity to assign to each market to maximize
the total profit under uncertain market conditions. This paper formulates the
bidding problem of the BESS as a Markov Decision Process, which enables the
BESS to participate in both the spot market and the FCAS market to maximize
profit. Then, Proximal Policy Optimization, a model-free deep reinforcement
learning algorithm, is employed to learn the optimal bidding strategy from the
dynamic environment of the energy market under a continuous bidding scale. The
proposed model is trained and validated using real-world historical data of the
Australian National Electricity Market. The results demonstrate that our
developed joint bidding strategy in both markets is significantly profitable
compared to individual markets.
