  Matrix-stepsized gradient descent algorithms have been shown to have superior
performance in non-convex optimization problems compared to their scalar
counterparts. The det-CGD algorithm, as introduced by Li et al. (2023),
leverages matrix stepsizes to perform compressed gradient descent for
non-convex objectives and matrix-smooth problems in a federated manner. The
authors establish the algorithm's convergence to a neighborhood of a weighted
stationarity point under a convex condition for the symmetric and
positive-definite matrix stepsize. In this paper, we propose two
variance-reduced versions of the det-CGD algorithm, incorporating MARINA and
DASHA methods. Notably, we establish theoretically and empirically, that
det-MARINA and det-DASHA outperform MARINA, DASHA and the distributed det-CGD
algorithms in terms of iteration and communication complexities.
