  Over the last two decades, the field of computational science has seen a
dramatic shift towards incorporating high-throughput computation and big-data
analysis as fundamental pillars of the scientific discovery process. This has
necessitated the development of tools and techniques to deal with the
generation, storage and processing of large amounts of data. In this work we
present an in-depth look at the workflow engine powering AiiDA, a widely
adopted, highly flexible and database-backed informatics infrastructure with an
emphasis on data reproducibility. We detail many of the design choices that
were made which were informed by several important goals: the ability to scale
from running on individual laptops up to high-performance supercomputers,
managing jobs with runtimes spanning from fractions of a second to weeks and
scaling up to thousands of jobs concurrently, and all this while maximising
robustness. In short, AiiDA aims to be a Swiss army knife for high-throughput
computational science. As well as the architecture, we outline important API
design choices made to give workflow writers a great deal of liberty whilst
guiding them towards writing robust and modular workflows, ultimately enabling
them to encode their scientific knowledge to the benefit of the wider
scientific community.
