  Scientific computing needs are growing dramatically with time and are
expanding in science domains that were previously not compute intensive. When
compute workflows spike well in excess of the capacity of their local compute
resource, capacity should be temporarily provisioned from somewhere else to
both meet deadlines and to increase scientific output. Public Clouds have
become an attractive option due to their ability to be provisioned with minimal
advance notice. The available capacity of cost-effective instances is not well
understood. This paper presents expanding the IceCube's production HTCondor
pool using cost-effective GPU instances in preemptible mode gathered from the
three major Cloud providers, namely Amazon Web Services, Microsoft Azure and
the Google Cloud Platform. Using this setup, we sustained for a whole workday
about 15k GPUs, corresponding to around 170 PFLOP32s, integrating over one
EFLOP32 hour worth of science output for a price tag of about $60k. In this
paper, we provide the reasoning behind Cloud instance selection, a description
of the setup and an analysis of the provisioned resources, as well as a short
description of the actual science output of the exercise.
