  libEnsemble is a Python-based toolkit for running dynamic ensembles,
developed as part of the DOE Exascale Computing Project. The toolkit utilizes a
unique generator--simulator--allocator paradigm, where generators produce input
for simulators, simulators evaluate those inputs, and allocators decide whether
and when a simulator or generator should be called. The generator steers the
ensemble based on simulation results. Generators may, for example, apply
methods for numerical optimization, machine learning, or statistical
calibration. libEnsemble communicates between a manager and workers. We
overview the unique characteristics of libEnsemble as well as current and
potential interoperability with other packages in the workflow ecosystem. We
highlight libEnsemble's dynamic resource features: libEnsemble can detect
system resources, such as available nodes, cores, and GPUs, and assign these in
a portable way. These features allow users to specify the number of processors
and GPUs required for each simulation; and resources will be automatically
assigned on a wide range of systems, including Frontier, Aurora, and
Perlmutter. Such ensembles can include multiple simulation types, some using
GPUs and others using only CPUs, sharing nodes for maximum efficiency. We also
describe the benefits of libEnsemble's generator--simulator coupling, which
easily exposes to the user the ability to cancel, and portably kill, running
simulations based on models that are updated with intermediate simulation
output. We demonstrate libEnsemble's capabilities, scalability, and scientific
impact via a Gaussian process surrogate training problem for the longitudinal
density profile at the exit of a plasma accelerator stage. The study uses gpCAM
for the surrogate model and employs either Wake-T or WarpX simulations,
highlighting efficient use of resources that can easily extend to exascale.
