  We present the implementation of a trust-region Newton algorithm ExaTron for
bound-constrained nonlinear programming problems, fully running on multiple
GPUs. Without data transfers between CPU and GPU, our implementation has
achieved the elimination of a major performance bottleneck under a memory-bound
situation, particularly when solving many small problems in batch. We discuss
the design principles and implementation details for our kernel function and
core operations. Different design choices are justified by numerical
experiments. By using the application of distributed control of alternating
current optimal power flow, where a large problem is decomposed into many
smaller nonlinear programs using a Lagrangian approach, we demonstrate
computational performance of ExaTron on the Summit supercomputer at Oak
RidgeNational Laboratory. Our numerical results show the linear scaling with
respect to the batch size and the number of GPUs and more than 35 times speedup
on 6 GPUs than on 40 CPUs available on a single node.
