  Extreme dynamic heterogeneity in high performance computing systems and the
convergence of traditional HPC with new simulation, analysis, and data science
approaches impose increasingly more complex requirements on resource and job
management software (RJMS). However, there is a paucity of RJMS techniques that
can solve key technical challenges associated with those new requirements,
particularly when they are coupled. In this paper, we propose a novel dynamic
and multi-level resource model approach to address three key well-known
challenges individually and in combination: i.e., 1) RJMS dynamism to
facilitate job and workflow adaptability, 2) integration of specialized
external resources (e.g. user-centric cloud bursting), and 3) scheduling cloud
orchestration framework tasks. The core idea is to combine a dynamic directed
graph resource model with fully hierarchical scheduling to provide a unified
solution to all three key challenges. Our empirical and analytical evaluations
of the solution using our prototype extension to Fluxion, a production
hierarchical graph-based scheduler, suggest that our unified solution can
significantly improve flexibility, performance and scalability across all three
problems in comparison to limited traditional approaches.
