  We propose an optimization algorithm called Hamiltonian Descent, which is a
direct counterpart of classical Hamiltonian Monte Carlo in sampling. We find
that Hamiltonian Descent for solving strongly convex quadratic problems
exhibits a novel update scheme that involves matrix-power-vector products. We
also propose Coordinate Hamiltonian Descent and its parallelizable variant,
which turns out to encapsulate the classical Gauss-Seidel method, Successive
Over-relaxation, Jacobi method, and more, for solving a linear system of
equations. The result not only offers a new perspective on these existing
algorithms but also leads to a broader class of update schemes that guarantee
the convergence.
