  We propose and solve a stochastic dynamic programming (DP) problem addressing
the optimal provision of regulation service reserves (RSR) by controlling
dynamic demand preferences in smart buildings. A major contribution over past
dynamic pricing work is that we pioneer the relaxation of static, uniformly
distributed utility of demand. In this paper we model explicitly the dynamics
of energy service preferences leading to a non-uniform and time varying
probability distribution of demand utility. More explicitly, we model active
and idle duty cycle appliances in a smart building as a closed queuing system
with price-controlled arrival rates into the active appliance queue. Focusing
on cooling appliances, we model the utility associated with the transition from
idle to active as a non-uniform time varying function. We (i) derive an
analytic characterization of the optimal policy and the differential cost
function, and (ii) prove optimal policy monotonicity and value function
convexity. These properties enable us to propose and implement a smart assisted
value iteration (AVI) algorithm and an approximate DP (ADP) that exploits
related functional approximations. Numerical results demonstrate the validity
of the solution techniques and the computational advantage of the proposed ADP
on realistic, large-state-space problems.
