  Utilities have introduced demand charges to encourage customers to reduce
their demand peaks, since a high peak may cause very high costs for both the
utility and the consumer. We herein study the bill minimization problem for
customers equipped with an energy storage device and a self-owned renewable
energy production. A model-free reinforcement learning algorithm is carefully
designed to reduce both the energy charge and the demand charge of the
consumer. The proposed algorithm does not need forecasting models for the
energy demand and the renewable energy production. The resulting controller can
be used online, and progressively improved with newly gathered data. The
algorithm is validated on real data from an office building of IFPEN Solaize
site. Numerical results show that our algorithm can reduce electricity bills
with both daily and monthly demand charges.
