  Hybrid main memory systems combine both performance and capacity advantages
from heterogeneous memory technologies. With larger capacities, higher
associativities, and finer granularities, hybrid memory systems currently
exhibit significant metadata storage and lookup overheads for flexibly
remapping data blocks between the two memory tiers. To alleviate the
inefficiencies of existing designs, we propose Trimma, the combination of a
multi-level metadata structure and an efficient metadata cache design. Trimma
uses a multi-level metadata table to only track truly necessary address remap
entries. The saved memory space is effectively utilized as extra DRAM cache
capacity to improve performance. Trimma also uses separate formats to store the
entries with non-identity and identity mappings. This improves the overall
remap cache hit rate, further boosting the performance. Trimma is transparent
to software and compatible with various types of hybrid memory systems. When
evaluated on a representative DDR4 + NVM hybrid memory system, Trimma achieves
up to 2.4$\times$ and on average 58.1\% speedup benefits, compared with a
state-of-the-art design that only leverages the unallocated fast memory space
for caching. Trimma addresses metadata management overheads and targets future
scalable large-scale hybrid memory architectures.
