  Complex scientific experiments from various domains are typically modeled as
workflows and executed on large-scale machines using a Parallel Workflow
Management System (WMS). Since such executions usually last for hours or days,
some WMSs provide user steering support, i.e., they allow users to run data
analyses and, depending on the results, adapt the workflows at runtime. A
challenge in the parallel execution control design is to manage workflow data
for efficient executions while enabling user steering support. Data access for
high scalability is typically transaction-oriented, while for data analysis, it
is online analytical-oriented so that managing such hybrid workloads makes the
challenge even harder. In this work, we present SchalaDB, an architecture with
a set of design principles and techniques based on distributed in-memory data
management for efficient workflow execution control and user steering. We
propose a distributed data design for scalable workflow task scheduling and
high availability driven by a parallel and distributed in-memory DBMS. To
evaluate our proposal, we develop d-Chiron, a WMS designed according to
SchalaDB's principles. We carry out an extensive experimental evaluation on an
HPC cluster with up to 960 computing cores. Among other analyses, we show that
even when running data analyses for user steering, SchalaDB's overhead is
negligible for workloads composed of hundreds of concurrent tasks on shared
data. Our results encourage workflow engine developers to follow a parallel and
distributed data-oriented approach not only for scheduling and monitoring but
also for user steering.
