  Numerical weather prediction (NWP) has proven to be computationally
challenging due to its inherent multiscale nature. Currently, the highest
resolution NWP models use a horizontal resolution of about 10km. In order to
increase the resolution of NWP models highly scalable atmospheric models are
needed.
  The Non-hydrostatic Unified Model of the Atmosphere (NUMA), developed by the
authors at the Naval Postgraduate School, was designed to achieve this purpose.
NUMA is used by the Naval Research Laboratory, Monterey as the engine inside
its next generation weather prediction system NEPTUNE. NUMA solves the fully
compressible Navier-Stokes equations by means of high-order Galerkin methods
(both spectral element as well as discontinuous Galerkin methods can be used).
Mesh generation is done using the p4est library. NUMA is capable of running
middle and upper atmosphere simulations since it does not make use of the
shallow-atmosphere approximation.
  This paper presents the performance analysis and optimization of the spectral
element version of NUMA. The performance at different optimization stages is
analyzed using a theoretical performance model as well as measurements via
hardware counters. Machine independent optimization is compared to machine
specific optimization using BG/Q vector intrinsics. By using vector intrinsics
the main computations reach 1.2 PFlops on the entire machine Mira (12% of the
theoretical peak performance). The paper also presents scalability studies for
two idealized test cases that are relevant for NWP applications. The
atmospheric model NUMA delivers an excellent strong scaling efficiency of 99%
on the entire supercomputer Mira using a mesh with 1.8 billion grid points.
This allows to run a global forecast of a baroclinic wave test case at 3km
uniform horizontal resolution and double precision within the time frame
required for operational weather prediction.
