  Modern computer systems are characterized by deep memory hierarchies,
composed of main memory, multiple layers of cache, and other specialized types
of memory. In parallel and distributed systems, additional memory layers are
added to this hierarchy. Achieving good performance for computational science
applications, in terms of execution time, depends on the efficient use of this
diverse and hierarchical memory. This paper revisits the use of space-filling
curves to specify the ordering in memory of data structures used in
representative scientific applications executing on parallel machines
containing clusters of multicore CPUs with attached GPUs. This work examines
the hypothesis that space-filling curves, such as Hilbert and Morton ordering,
can improve data locality and hence result in more efficient data movement than
row or column-based orderings. First, performance results are presented that
show for what application parameterizations and machine characteristics this is
the case, and are interpreted in terms of how an application interacts with the
computer hardware and low-level software. This research particularly focuses on
the use of stencil-based applications that form the basis of many scientific
computations. Second, how space-filling curves impact data sharing in
nearest-neighbour and stencil-based codes is considered.
