  HEPCloud is rapidly becoming the primary system for provisioning compute
resources for all Fermilab-affiliated experiments. In order to reliably meet
the peak demands of the next generation of High Energy Physics experiments,
Fermilab must plan to elastically expand its computational capabilities to
cover the forecasted need. Commercial cloud and allocation-based High
Performance Computing (HPC) resources both have explicit and implicit costs
that must be considered when deciding when to provision these resources, and at
which scale. In order to support such provisioning in a manner consistent with
organizational business rules and budget constraints, we have developed a
modular intelligent decision support system (IDSS) to aid in the automatic
provisioning of resources spanning multiple cloud providers, multiple HPC
centers, and grid computing federations. In this paper, we discuss the goals
and architecture of the HEPCloud Facility, the architecture of the IDSS, and
our early experience in using the IDSS for automated facility expansion both at
Fermi and Brookhaven National Laboratory.
