  Solving partial differential equations with deep learning makes it possible
to reduce simulation times by multiple orders of magnitude and unlock
scientific methods that typically rely on large numbers of sequential
simulations, such as optimization and uncertainty quantification. Two of the
largest challenges of adopting scientific AI for industrial problem settings is
that training datasets must be simulated in advance and that neural networks
for solving large-scale PDEs exceed the memory capabilities of current GPUs. We
introduce a distributed programming API in the Julia language for simulating
training data in parallel on the cloud and without requiring users to manage
the underlying HPC infrastructure. In addition, we show that model-parallel
deep learning based on domain decomposition allows us to scale neural networks
for solving PDEs to commercial-scale problem settings and achieve above 90%
parallel efficiency. Combining our cloud API for training data generation and
model-parallel deep learning, we train large-scale neural networks for solving
the 3D Navier-Stokes equation and simulating 3D CO2 flow in porous media. For
the CO2 example, we simulate a training dataset based on a commercial carbon
capture and storage (CCS) project and train a neural network for CO2 flow
simulation on a 3D grid with over 2 million cells that is 5 orders of
magnitudes faster than a conventional numerical simulator and 3,200 times
cheaper.
