  Resource selection and task placement for distributed execution poses
conceptual and implementation difficulties. Although resource selection and
task placement are at the core of many tools and workflow systems, the methods
are ad hoc rather than being based on models. Consequently, partial and
non-interoperable implementations proliferate. We address both the conceptual
and implementation difficulties by experimentally characterizing diverse
modalities of resource selection and task placement. We compare the
architectures and capabilities of two systems: the AIMES middleware and Swift
workflow scripting language and runtime. We integrate these systems to enable
the distributed execution of Swift workflows on Pilot-Jobs managed by the AIMES
middleware. Our experiments characterize and compare alternative execution
strategies by measuring the time to completion of heterogeneous uncoupled
workloads executed at diverse scale and on multiple resources. We measure the
adverse effects of pilot fragmentation and early binding of tasks to resources
and the benefits of backfill scheduling across pilots on multiple resources. We
then use this insight to execute a multi-stage workflow across five
production-grade resources. We discuss the importance and implications for
other tools and workflow systems.
