  Leadership supercomputers feature a diversity of storage, from node-local
persistent memory and NVMe SSDs to network-interconnected flash memory and HDD.
Memory mapping files on different tiers of storage provides a uniform interface
in applications. However, system-wide services like mmap are optimized for
generality and lack flexibility for enabling application-specific
optimizations. In this work, we present Umap to enable user-space page
management that can be easily adapted to access patterns in applications and
storage characteristics. Umap uses the userfaultfd mechanism to handle page
faults in multi-threaded applications efficiently. By providing a data object
abstraction layer, Umap is extensible to support various backing stores. The
design of Umap supports dynamic load balancing and I/O decoupling for scalable
performance. Umap also uses application hints to improve the selection of
caching, prefetching, and eviction policies. We evaluate Umap in five
benchmarks and real applications on two systems. Our results show that
leveraging application knowledge for page management could substantially
improve performance. On average, Umap achieved 1.25 to 2.5 times improvement
using the adapted configurations compared to the system service.
