  We study ways to accelerate greedy coordinate descent in theory and in
practice, where "accelerate" refers either to $O(1/k^2)$ convergence in theory,
in practice, or both. We introduce and study two algorithms: Accelerated
Semi-Greedy Coordinate Descent (ASCD) and Accelerated Greedy Coordinate Descent
(AGCD). While ASCD takes greedy steps in the $x$-updates and randomized steps
in the $z$-updates, AGCD is a straightforward extension of standard greedy
coordinate descent that only takes greedy steps. On the theory side, our main
results are for ASCD: we show that ASCD achieves $O(1/k^2)$ convergence, and it
also achieves accelerated linear convergence for strongly convex functions. On
the empirical side, we observe that both AGCD and ASCD outperform Accelerated
Randomized Coordinate Descent on a variety of instances. In particular, we note
that AGCD significantly outperforms the other accelerated coordinate descent
methods in numerical tests, in spite of a lack of theoretical guarantees for
this method. To complement the empirical study of AGCD, we present a Lyapunov
energy function argument that points to an explanation for why a direct
extension of the acceleration proof for AGCD does not work; and we also
introduce a technical condition under which AGCD is guaranteed to have
accelerated convergence. Last of all, we confirm that this technical condition
holds in our empirical study.
