  We present the acceleration of an IMplicit-EXplicit (IMEX) non-hydrostatic
atmospheric model on manycore processors such as GPUs and Intel's MIC
architecture. IMEX time integration methods sidestep the constraint imposed by
the Courant-Friedrichs-Lewy condition on explicit methods through corrective
implicit solves within each time step. In this work, we implement and evaluate
the performance of IMEX on manycore processors relative to explicit methods.
Using 3D-IMEX at Courant number C=15 , we obtained a speedup of about 4X
relative to an explicit time stepping method run with the maximum allowable
C=1. In addition, we demonstrate a much larger speedup of 100X at C=150 using
1D-IMEX due to the unconditional stability of the method in the vertical
direction. Several improvements on the IMEX procedure were necessary in order
to outperform our results with explicit methods: a) reducing the number of
degrees of freedom of the IMEX formulation by forming the Schur complement; b)
formulating a horizontally-explicit vertically-implicit (HEVI) 1D-IMEX scheme
that has a lower workload and potentially better scalability than 3D-IMEX; c)
using high-order polynomial preconditioners to reduce the condition number of
the resulting system; d) using a direct solver for the 1D-IMEX method by
performing and storing LU factorizations once to obtain a constant cost for any
Courant number. Without all of these improvements, explicit time integration
methods turned out to be difficult to beat. We discuss in detail the IMEX
infrastructure required for formulating and implementing efficient methods on
manycore processors. Finally, we validate our results with standard benchmark
problems in NWP and evaluate the performance and scalability of the IMEX method
using up to 4192 GPUs and 16 Knights Landing processors.
