  The power grid is a complex and vital system that necessitates careful
reliability management. Managing the grid is a difficult problem with multiple
time scales of decision making and stochastic behavior due to renewable energy
generations, variable demand and unplanned outages. Solving this problem in the
face of uncertainty requires a new methodology with tractable algorithms. In
this work, we introduce a new model for hierarchical decision making in complex
systems. We apply reinforcement learning (RL) methods to learn a proxy, i.e., a
level of abstraction, for real-time power grid reliability. We devise an
algorithm that alternates between slow time-scale policy improvement, and fast
time-scale value function approximation. We compare our results to prevailing
heuristics, and show the strength of our method.
