  The parallel annealing method is one of the promising approaches for large
scale simulations as potentially scalable on any parallel architecture. We
present an implementation of the algorithm on the hybrid program architecture
combining CUDA and MPI. The problem is to keep all general-purpose graphics
processing unit devices as busy as possible redistributing replicas and to do
that efficiently. We provide details of the testing on Intel Skylake/Nvidia
V100 based hardware running in parallel more than two million replicas of the
Ising model sample. The results are quite optimistic because the acceleration
grows toward the perfect line with the growing complexity of the simulated
system.
