  Parallel programmers face the often irreconcilable goals of programmability
and performance. HPC systems use distributed memory for scalability, thereby
sacrificing the programmability advantages of shared memory programming models.
Furthermore, the rapid adoption of heterogeneous architectures, often with
non-cache-coherent memory systems, has further increased the challenge of
supporting shared memory programming models. Our primary objective is to define
a memory consistency model that presents the familiar thread-based shared
memory programming model, but allows good application performance on
non-cache-coherent systems, including distributed memory clusters and
accelerator-based systems. We propose regional consistency (RegC), a new
consistency model that achieves this objective. Results on up to 256 processors
for representative benchmarks demonstrate the potential of RegC in the context
of our prototype distributed shared memory system.
