  To improve system performance, modern operating systems (OSes) often
undertake activities that require modification of virtual-to-physical page
translation mappings. For example, the OS may migrate data between physical
frames to defragment memory and enable superpages. The OS may migrate pages of
data between heterogeneous memory devices. We refer to all such activities as
page remappings. Unfortunately, page remappings are expensive. We show that
translation coherence is a major culprit and that systems employing
virtualization are especially badly affected by their overheads. In response,
we propose hardware translation invalidation and coherence or HATRIC, a readily
implementable hardware mechanism to piggyback translation coherence atop
existing cache coherence protocols. We perform detailed studies using KVM-based
virtualization, showing that HATRIC achieves up to 30% performance and 10%
energy benefits, for per-CPU area overheads of 2%. We also quantify HATRIC's
benefits on systems running Xen and find up to 33% performance improvements.
