  Memory management operations that modify page-tables, typically performed
during memory allocation/deallocation, are infamous for their poor performance
in highly threaded applications, largely due to process-wide TLB shootdowns
that the OS must issue due to the lack of hardware support for TLB coherence.
We study these operations in NUMA settings, where we observe up to 40x overhead
for basic operations such as munmap or mprotect. The overhead further increases
if page-table replication is used, where complete coherent copies of the
page-tables are maintained across all NUMA nodes. While eager system-wide
replication is extremely effective at localizing page-table reads during
address translation, we find that it creates additional penalties upon any
page-table changes due to the need to maintain all replicas coherent.
  In this paper, we propose a novel page-table management mechanism, called
numaPTE, to enable transparent, on-demand, and partial page-table replication
across NUMA nodes in order to perform address translation locally, while
avoiding the overheads and scalability issues of system-wide full page-table
replication. We then show that numaPTE's precise knowledge of page-table
sharers can be leveraged to significantly reduce the number of TLB shootdowns
issued upon any memory-management operation. As a result, numaPTE not only
avoids replication-related slowdowns, but also provides significant speedup
over the baseline on memory allocation/deallocation and access control
operations. We implement numaPTEin Linux on x86_64, evaluate it on 4- and
8-socket systems, and show that numaPTE achieves the full benefits of eager
page-table replication on a wide range of applications, while also achieving a
12% and 36% runtime improvement on Webserver and Memcached respectively due to
a significant reduction in TLB shootdowns.
