  Cognitive simulation (CogSim) is an important and emerging workflow for HPC
scientific exploration and scientific machine learning (SciML). One challenging
workload for CogSim is the replacement of one component in a complex physical
simulation with a fast, learned, surrogate model that is "inside" of the
computational loop. The execution of this in-the-loop inference is particularly
challenging because it requires frequent inference across multiple possible
target models, can be on the simulation's critical path (latency bound), is
subject to requests from multiple MPI ranks, and typically contains a small
number of samples per request. In this paper we explore the use of large,
dedicated Deep Learning / AI accelerators that are disaggregated from compute
nodes for this CogSim workload. We compare the trade-offs of using these
accelerators versus the node-local GPU accelerators on leadership-class HPC
systems.
