  Runtime scheduling and workflow systems are an increasingly popular
algorithmic component in HPC because they allow full system utilization with
relaxed synchronization requirements. There are so many special-purpose tools
for task scheduling, one might wonder why more are needed. Use cases seen on
the Summit supercomputer needed better integration with MPI and greater
flexibility in job launch configurations. Preparation, execution, and analysis
of computational chemistry simulations at the scale of tens of thousands of
processors revealed three distinct workflow patterns. A separate job scheduler
was implemented for each one using extremely simple and robust designs:
file-based, task-list based, and bulk-synchronous. Comparing to existing
methods shows unique benefits of this work, including simplicity of design,
suitability for HPC centers, short startup time, and well-understood per-task
overhead. All three new tools have been shown to scale to full utilization of
Summit, and have been made publicly available with tests and documentation.
This work presents a complete characterization of the minimum effective task
granularity for efficient scheduler usage scenarios. These schedulers have the
same bottlenecks, and hence similar task granularities as those reported for
existing tools following comparable paradigms.
