  Task graphs provide a simple way to describe scientific workflows (sets of
tasks with dependencies) that can be executed on both HPC clusters and in the
cloud. An important aspect of executing such graphs is the used scheduling
algorithm. Many scheduling heuristics have been proposed in existing works;
nevertheless, they are often tested in oversimplified environments. We provide
an extensible simulation environment designed for prototyping and benchmarking
task schedulers, which contains implementations of various scheduling
algorithms and is open-sourced, in order to be fully reproducible. We use this
environment to perform a comprehensive analysis of workflow scheduling
algorithms with a focus on quantifying the effect of scheduling challenges that
have so far been mostly neglected, such as delays between scheduler invocations
or partially unknown task durations. Our results indicate that network models
used by many previous works might produce results that are off by an order of
magnitude in comparison to a more realistic model. Additionally, we show that
certain implementation details of scheduling algorithms which are often
neglected can have a large effect on the scheduler's performance, and they
should thus be described in great detail to enable proper evaluation.
