  There are different strategies for training neural networks (NNs) as
subgrid-scale parameterizations. Here, we use a 1D model of the quasi-biennial
oscillation (QBO) and gravity wave (GW) parameterizations as testbeds. A
12-layer convolutional NN that predicts GW forcings for given wind profiles,
when trained offline in a big-data regime (100-years), produces realistic QBOs
once coupled to the 1D model. In contrast, offline training of this NN in a
small-data regime (18-months) yields unrealistic QBOs. However, online
re-training of just two layers of this NN using ensemble Kalman inversion and
only time-averaged QBO statistics leads to parameterizations that yield
realistic QBOs. Fourier analysis of these three NNs' kernels suggests why/how
re-training works and reveals that these NNs primarily learn low-pass,
high-pass, and a combination of band-pass filters, consistent with the
importance of both local and non-local dynamics in GW propagation/dissipation.
These findings/strategies apply to data-driven parameterizations of other
climate processes generally.
