  We present a parallel implementation of a direct solver for the Poisson's
equation on extreme-scale supercomputers with accelerators. We introduce a
chunked-pencil decomposition as the domain-decomposition strategy to distribute
work among processing elements to achieve superior scalability at large number
of accelerators. Chunked-pencil decomposition enables overlapping nodal
communication and data transfer between the central processing units (CPUs) and
the graphics processing units (GPUs). Second, it improves data locality by
keeping neighboring elements in adjacent memory locations. Third, it allows
usage of shared-memory for certain segments of the algorithm when possible, and
last but not least, it enables contiguous message transfer among the nodes. Two
different communication patterns are designed. The fist pattern aims to fully
overlap the communication with data transfer and designed for speedup of
overall turnaround time, whereas the second method concentrates on low memory
usage and is more network friendly for computations at extreme scale. To ensure
software portability, we interleave OpenACC with MPI in the software. The
numerical solution and its formal second order of accuracy is verified using
method of manufactured solutions for various combinations of boundary
conditions. Weak scaling analysis is performed using up to 1.1 trillion
Cartesian mesh points using 16384 GPUs on a petascale leadership class
supercomputer.
