  In this paper, we propose GT-GDA, a distributed optimization method to solve
saddle point problems of the form: $\min_{\mathbf{x}} \max_{\mathbf{y}}
\{F(\mathbf{x},\mathbf{y}) :=G(\mathbf{x}) + \langle \mathbf{y}, \overline{P}
\mathbf{x} \rangle - H(\mathbf{y})\}$, where the functions $G(\cdot)$,
$H(\cdot)$, and the the coupling matrix $\overline{P}$ are distributed over a
strongly connected network of nodes. GT-GDA is a first-order method that uses
gradient tracking to eliminate the dissimilarity caused by heterogeneous data
distribution among the nodes. In the most general form, GT-GDA includes a
consensus over the local coupling matrices to achieve the optimal (unique)
saddle point, however, at the expense of increased communication. To avoid
this, we propose a more efficient variant GT-GDA-Lite that does not incur the
additional communication and analyze its convergence in various scenarios. We
show that GT-GDA converges linearly to the unique saddle point solution when
$G(\cdot)$ is smooth and convex, $H(\cdot)$ is smooth and strongly convex, and
the global coupling matrix $\overline{P}$ has full column rank. We further
characterize the regime under which GT-GDA exhibits a network
topology-independent convergence behavior. We next show the linear convergence
of GT-GDA to an error around the unique saddle point, which goes to zero when
the coupling cost ${\langle \mathbf y, \overline{P} \mathbf x \rangle}$ is
common to all nodes, or when $G(\cdot)$ and $H(\cdot)$ are quadratic. Numerical
experiments illustrate the convergence properties and importance of GT-GDA and
GT-GDA-Lite for several applications.
