  Developing surrogate geophysical models from data is a key research topic in
atmospheric and oceanic modeling because of the large computational costs
associated with numerical simulation methods. Researchers have started applying
a wide range of machine learning models, in particular neural networks, to
geophysical data for forecasting without these constraints. Constructing neural
networks for forecasting such data is nontrivial, however, and often requires
trial and error. To address these limitations, we focus on developing
proper-orthogonal-decomposition-based long short-term memory networks
(POD-LSTMs). We develop a scalable neural architecture search for generating
stacked LSTMs to forecast temperature in the NOAA Optimum Interpolation
Sea-Surface Temperature data set. Our approach identifies POD-LSTMs that are
superior to manually designed variants and baseline time-series prediction
methods. We also assess the scalability of different architecture search
strategies on up to 512 Intel Knights Landing nodes of the Theta supercomputer
at the Argonne Leadership Computing Facility.
