  Multilayer graphs are commonly used for representing different relations
between entities and handling heterogeneous data processing tasks. Non-standard
multilayer graph clustering methods are needed for assigning clusters to a
common multilayer node set and for combining information from each layer. This
paper presents a multilayer spectral graph clustering (SGC) framework that
performs convex layer aggregation. Under a multilayer signal plus noise model,
we provide a phase transition analysis of clustering reliability. Moreover, we
use the phase transition criterion to propose a multilayer iterative model
order selection algorithm (MIMOSA) for multilayer SGC, which features automated
cluster assignment and layer weight adaptation, and provides statistical
clustering reliability guarantees. Numerical simulations on synthetic
multilayer graphs verify the phase transition analysis, and experiments on
real-world multilayer graphs show that MIMOSA is competitive or better than
other clustering methods.
