  To accelerate the solution of large eigenvalue problems arising from
many-body calculations in nuclear physics on distributed-memory parallel
systems equipped with general-purpose Graphic Processing Units (GPUs), we
modified a previously developed hybrid MPI/OpenMP implementation of an
eigensolver written in FORTRAN 90 by using an OpenACC directives based
programming model. Such an approach requires making minimal changes to the
original code and enables a smooth migration of large-scale nuclear structure
simulations from a distributed-memory many-core CPU system to a distributed GPU
system. However, in order to make the OpenACC based eigensolver run efficiently
on GPUs, we need to take into account the architectural differences between a
many-core CPU and a GPU device. Consequently, the optimal way to insert OpenACC
directives may be different from the original way of inserting OpenMP
directives. We point out these differences in the implementation of sparse
matrix-matrix multiplications (SpMM), which constitutes the main cost of the
eigensolver, as well as other differences in the preconditioning step and dense
linear algebra operations. We compare the performance of the OpenACC based
implementation executed on multiple GPUs with the performance on
distributed-memory many-core CPUs, and demonstrate significant speedup achieved
on GPUs compared to the on-node performance of a many-core CPU. We also show
that the overall performance improvement of the eigensolver on multiple GPUs is
more modest due to the communication overhead among different MPI ranks.
