  The conventional control paradigm for a heat pump with a less efficient
auxiliary heating element is to keep its temperature set point constant during
the day. This constant temperature set point ensures that the heat pump
operates in its more efficient heat-pump mode and minimizes the risk of
activating the less efficient auxiliary heating element. As an alternative to a
constant set-point strategy, this paper proposes a learning agent for a
thermostat with a set-back strategy. This set-back strategy relaxes the
set-point temperature during convenient moments, e.g. when the occupants are
not at home. Finding an optimal set-back strategy requires solving a sequential
decision-making process under uncertainty, which presents two challenges. A
first challenge is that for most residential buildings a description of the
thermal characteristics of the building is unavailable and challenging to
obtain. A second challenge is that the relevant information on the state, i.e.
the building envelope, cannot be measured by the learning agent. In order to
overcome these two challenges, our paper proposes an auto-encoder coupled with
a batch reinforcement learning technique. The proposed approach is validated
for two building types with different thermal characteristics for heating in
the winter and cooling in the summer. The simulation results indicate that the
proposed learning agent can reduce the energy consumption by 4-9% during 100
winter days and by 9-11% during 80 summer days compared to the conventional
constant set-point strategy
