  The problem of efficient modulation classification (MC) in multiple-input
multiple-output (MIMO) systems is considered. Per-layer likelihood-based MC is
proposed by employing subspace decomposition to partially decouple the
transmitted streams. When detecting the modulation type of the stream of
interest, a dense constellation is assumed on all remaining streams. The
proposed classifier outperforms existing MC schemes at a lower complexity cost,
and can be efficiently implemented in the context of joint MC and subspace data
detection.
