  This paper introduces a novel framework called Mode-wise Principal Subspace
Pursuit (MOP-UP) to extract hidden variations in both the row and column
dimensions for matrix data. To enhance the understanding of the framework, we
introduce a class of matrix-variate spiked covariance models that serve as
inspiration for the development of the MOP-UP algorithm. The MOP-UP algorithm
consists of two steps: Average Subspace Capture (ASC) and Alternating
Projection (AP). These steps are specifically designed to capture the row-wise
and column-wise dimension-reduced subspaces which contain the most informative
features of the data. ASC utilizes a novel average projection operator as
initialization and achieves exact recovery in the noiseless setting. We analyze
the convergence and non-asymptotic error bounds of MOP-UP, introducing a
blockwise matrix eigenvalue perturbation bound that proves the desired bound,
where classic perturbation bounds fail. The effectiveness and practical merits
of the proposed framework are demonstrated through experiments on both
simulated and real datasets. Lastly, we discuss generalizations of our approach
to higher-order data.
