  A scalable algorithm for solving compact banded linear systems on distributed
memory architectures is presented. The proposed method factorizes the original
system into two levels of memory hierarchies, and solves it using parallel
cyclic reduction on both distributed and shared memory. This method has a lower
communication footprint across distributed memory partitions compared to
conventional algorithms involving data transpose or re-partitioning. The
algorithm developed in this work is generalized to cyclic compact banded
systems with flexible data decompositions. For cyclic compact banded systems,
the method is a direct solver with a deterministic operation and communication
counts depending on the matrix size, its bandwidth, and the partition strategy.
The implementation and runtime configuration details are discussed for
performance optimization. Scalability is demonstrated on the linear solver as
well as on a representative fluid mechanics application problem, in which the
dominant computational cost is solving the cyclic tridiagonal linear systems of
compact numerical schemes on a 3D periodic domain. The algorithm is
particularly useful for solving the linear systems arising from the application
of compact finite difference operators to a wide range of partial differential
equation problems, such as but not limited to the numerical simulations of
compressible turbulent flows, aeroacoustics, elastic-plastic wave propagation,
and electromagnetics. It alleviates obstacles to their use on modern high
performance computing hardware, where memory and computational power are
distributed across nodes with multi-threaded processing units.
