  The lattice Boltzmann method exhibits excellent scalability on current
supercomputing systems and has thus increasingly become an alternative method
for large-scale non-stationary flow simulations, reaching up to a trillion grid
nodes. Additionally, grid refinement can lead to substantial savings in memory
and compute time. These saving, however, come at the cost of much more complex
data structures and algorithms. In particular, the interface between subdomains
with different grid sizes must receive special treatment. In this article, we
present parallel algorithms, distributed data structures, and communication
routines that are implemented in the software framework waLBerla in order to
support large-scale, massively parallel lattice Boltzmann-based simulations on
non-uniform grids. Additionally, we evaluate the performance of our approach on
two current petascale supercomputers. On an IBM Blue Gene/Q system, the largest
weak scaling benchmarks with refined grids are executed with almost two million
threads, demonstrating not only near-perfect scalability but also an absolute
performance of close to a trillion lattice Boltzmann cell updates per second.
On an Intel-based system, the strong scaling of a simulation with refined grids
and a total of more than 8.5 million cells is demonstrated to reach a
performance of less than one millisecond per time step. This enables
simulations with complex, non-uniform grids and four million time steps per
hour compute time.
