  We present $\texttt{SIMULATeQCD}$, HotQCD's software for performing lattice
QCD calculations on GPUs. Started in late 2017 and intended as a full
replacement of the previous single GPU lattice QCD code used by the HotQCD
collaboration, our software has been developed into an extensive framework for
lattice QCD calculations distributed on multiple GPUs over many compute nodes.
The code is built on C++, CUDA, and MPI and leverages modern C++ language
features to provide high-level data structures, objects, and algorithms that
allow users to express lattice QCD calculations in an intuitive way without
sacrificing performance. Implemented algorithms range from gradient flow,
correlator measurements, and mixed precision conjugate gradient solvers all the
way to full HISQ gauge field configuration generation using RHMC. After
successful deployment in large-scale computing projects, we want to share the
result of our efforts with the lattice QCD community by making it publicly
available. In these proceedings, we will present some of the key features of
our code, demonstrate its ease of use, and show benchmarks of performance
critical kernels on state-of-the-art supercomputers.
