  We consider the problem of demand-side energy management, where each
household is equipped with a smart meter that is able to schedule home
appliances online. The goal is to minimize the overall cost under a real-time
pricing scheme. While previous works have introduced centralized approaches in
which the scheduling algorithm has full observability, we propose the
formulation of a smart grid environment as a Markov game. Each household is a
decentralized agent with partial observability, which allows scalability and
privacy-preservation in a realistic setting. The grid operator produces a price
signal that varies with the energy demand. We propose an extension to a
multi-agent, deep actor-critic algorithm to address partial observability and
the perceived non-stationarity of the environment from the agent's viewpoint.
This algorithm learns a centralized critic that coordinates training of
decentralized agents. Our approach thus uses centralized learning but
decentralized execution. Simulation results show that our online deep
reinforcement learning method can reduce both the peak-to-average ratio of
total energy consumed and the cost of electricity for all households based
purely on instantaneous observations and a price signal.
