  This paper introduces DEM-Engine, a new submodule of Project Chrono, that is
designed to carry out Discrete Element Method (DEM) simulations. Based on
spherical primitive shapes, DEM-Engine can simulate polydisperse granular
materials and handle complex shapes generated as assemblies of primitives,
referred to as clumps. DEM-Engine has a multi-tier parallelized structure that
is optimized to operate simultaneously on two GPUs. The code uses
custom-defined data types to reduce memory footprint and increase bandwidth. A
novel "delayed contact detection" algorithm allows the decoupling of the
contact detection and force computation, thus splitting the workload into two
asynchronous GPU streams. DEM-Engine uses just-in-time compilation to support
user-defined contact force models. This paper discusses its C++ and Python
interfaces and presents a variety of numerical tests, in which impact forces,
complex-shaped particle flows, and a custom force model are validated
considering well-known benchmark cases. Additionally, the full potential of the
simulator is demonstrated for the investigation of extraterrestrial rover
mobility on granular terrain. The chosen case study demonstrates that
large-scale co-simulations (comprising 11 million elements) spanning 15
seconds, in conjunction with an external multi-body dynamics system, can be
efficiently executed within a day. Lastly, a performance test suggests that
DEM-Engine displays linear scaling up to 150 million elements on two NVIDIA
A100 GPUs.
