  We consider the convex-concave saddle point problem $\min_{x}\max_{y}
f(x)+y^\top A x-g(y)$ where $f$ is smooth and convex and $g$ is smooth and
strongly convex. We prove that if the coupling matrix $A$ has full column rank,
the vanilla primal-dual gradient method can achieve linear convergence even if
$f$ is not strongly convex. Our result generalizes previous work which either
requires $f$ and $g$ to be quadratic functions or requires proximal mappings
for both $f$ and $g$. We adopt a novel analysis technique that in each
iteration uses a "ghost" update as a reference, and show that the iterates in
the primal-dual gradient method converge to this "ghost" sequence. Using the
same technique we further give an analysis for the primal-dual stochastic
variance reduced gradient (SVRG) method for convex-concave saddle point
problems with a finite-sum structure.
