  Because of increasing amounts of intermittent and distributed generators in
power systems, many demand response programs have been developed to schedule
flexible energy consumption. However, proper benchmarks for comparing these
methods are lacking, especially when decisions are made based on information
which is repeatedly updated. This paper presents a new benchmarking tool
designed to bridge this gap. Surveys that classify flexibility planning
algorithms were an input to define the benchmarking standards. The framework
can be used for different objectives and under diverse conditions faced by
electricity energy stakeholders interested in flexibility scheduling
algorithms. It includes a simulation environment that captures the evolution of
look--ahead information, which enables comparing online planning and scheduling
algorithms. The benchmarking tool is used to test seven planning algorithms
measuring their performance under uncertain market conditions. The results show
the importance of online decision making, the influence of data quality on the
algorithm performance, the benefit of using robust and stochastic programming
approaches, and the necessity of trustworthy benchmarking.
