  We consider a class of non-smooth strongly convex-strongly concave saddle
point problems in a decentralized setting without a central server. To solve a
consensus formulation of problems in this class, we develop an inexact primal
dual hybrid gradient (inexact PDHG) procedure that allows generic gradient
computation oracles to update the primal and dual variables. We first
investigate the performance of inexact PDHG with stochastic variance reduction
gradient (SVRG) oracle. Our numerical study uncovers a significant phenomenon
of initial conservative progress of iterates of IPDHG with SVRG oracle. To
tackle this, we develop a simple and effective switching idea, where a
generalized stochastic gradient (GSG) computation oracle is employed to hasten
the iterates' progress to a saddle point solution during the initial phase of
updates, followed by a switch to the SVRG oracle at an appropriate juncture.
The proposed algorithm is named Decentralized Proximal Switching Stochastic
Gradient method with Compression (C-DPSSG), and is proven to converge to an
$\epsilon$-accurate saddle point solution with linear rate. Apart from
delivering highly accurate solutions, our study reveals that utilizing the best
convergence phases of GSG and SVRG oracles makes C-DPSSG well suited for
obtaining solutions of low/medium accuracy faster, useful for certain
applications. Numerical experiments on two benchmark machine learning
applications show C-DPSSG's competitive performance which validate our
theoretical findings. The codes used in the experiments can be found
\href{https://github.com/chhavisharma123/C-DPSSG-CDC2023}{here}.
