  In this paper we study K-polystability of arbitrary (possibly non-projective)
compact K\"ahler manifolds admitting holomorphic vector fields. As a main
result, we show that existence of a constant scalar curvature K\"ahler (cscK)
metric implies 'geodesic K-polystability', in a sense that is expected to be
equivalent to K-polystability in general. In particular, in the spirit of an
expectation of Chen-Tang we show that geodesic K-polystability implies
algebraic K-polystability for polarized manifolds. Hence our main result
recovers a possibly stronger version of results of Berman-Darvas-Lu in this
case. As a key part of the proof we also study subgeodesic rays with
singularity type prescribed by singular test configurations, and prove a result
on asymptotics of the K-energy functional along such rays.
  In an appendix by R. Dervan it is moreover deduced that geodesic
K-polystability implies equivariant K-polystability. This improves upon
previous work and proves that existence of a cscK (or extremal) K\"ahler metric
implies equivariant K-polystability (resp. relative K-stability).
