  Demand flexibility plays a vital role in maintaining grid balance, reducing
peak demand, and saving customers' energy bills. Given their highly shiftable
load and significant contribution to a building's energy consumption, Heating,
Ventilation, and Air Conditioning (HVAC) systems can provide valuable demand
flexibility to the power systems by adjusting their energy consumption in
response to electricity price and power system needs. To exploit this
flexibility in both operation time and power, it is imperative to accurately
model and aggregate the load flexibility of a large population of HVAC systems
as well as designing effective control algorithms. In this paper, we tackle the
curse of dimensionality issue in modeling and control by utilizing the concept
of laxity to quantify the emergency level of each HVAC operation request. We
further propose a two-level approach to address energy optimization for a large
population of HVAC systems. The lower level involves an aggregator to aggregate
HVAC load laxity information and use least-laxity-first (LLF) rule to allocate
real-time power for individual HVAC systems based on the controller's total
power. Due to the complex and uncertain nature of HVAC systems, we leverage a
reinforcement learning (RL)-based controller to schedule the total power based
on the aggregated laxity information and electricity price. We evaluate the
temperature control and energy cost saving performance of a large-scale group
of HVAC systems in both single-zone and multi-zone scenarios, under varying
climate and electricity market conditions. The experiment results indicate that
proposed approach outperforms the centralized methods in the majority of test
scenarios, and performs comparably to model-based method in some scenarios.
