  Electric water heaters have the ability to store energy in their water buffer
without impacting the comfort of the end user. This feature makes them a prime
candidate for residential demand response. However, the stochastic and
nonlinear dynamics of electric water heaters, makes it challenging to harness
their flexibility. Driven by this challenge, this paper formulates the
underlying sequential decision-making problem as a Markov decision process and
uses techniques from reinforcement learning. Specifically, we apply an
auto-encoder network to find a compact feature representation of the sensor
measurements, which helps to mitigate the curse of dimensionality. A wellknown
batch reinforcement learning technique, fitted Q-iteration, is used to find a
control policy, given this feature representation. In a simulation-based
experiment using an electric water heater with 50 temperature sensors, the
proposed method was able to achieve good policies much faster than when using
the full state information. In a lab experiment, we apply fitted Q-iteration to
an electric water heater with eight temperature sensors. Further reducing the
state vector did not improve the results of fitted Q-iteration. The results of
the lab experiment, spanning 40 days, indicate that compared to a thermostat
controller, the presented approach was able to reduce the total cost of energy
consumption of the electric water heater by 15%.
