  In a modern GPU architecture, all threads within a warp execute the same
instruction in lockstep. For a memory instruction, this can lead to memory
divergence: the memory requests for some threads are serviced early, while the
remaining requests incur long latencies. This divergence stalls the warp, as it
cannot execute the next instruction until all requests from the current
instruction complete. In this work, we make three new observations. First,
GPGPU warps exhibit heterogeneous memory divergence behavior at the shared
cache: some warps have most of their requests hit in the cache, while other
warps see most of their request miss. Second, a warp retains the same
divergence behavior for long periods of execution. Third, requests going to the
shared cache can incur queuing delays as large as hundreds of cycles,
exacerbating the effects of memory divergence. We propose a set of techniques,
collectively called Memory Divergence Correction (MeDiC), that reduce the
negative performance impact of memory divergence and cache queuing. MeDiC
delivers an average speedup of 21.8%, and 20.1% higher energy efficiency, over
a state-of-the-art GPU cache management mechanism across 15 different GPGPU
applications.
