  In large-scale distributed machine learning, recent works have studied the
effects of compressing gradients in stochastic optimization to alleviate the
communication bottleneck. These works have collectively revealed that
stochastic gradient descent (SGD) is robust to structured perturbations such as
quantization, sparsification, and delays. Perhaps surprisingly, despite the
surge of interest in multi-agent reinforcement learning, almost nothing is
known about the analogous question: Are common reinforcement learning (RL)
algorithms also robust to similar perturbations? We investigate this question
by studying a variant of the classical temporal difference (TD) learning
algorithm with a perturbed update direction, where a general compression
operator is used to model the perturbation. Our work makes three important
technical contributions. First, we prove that compressed TD algorithms, coupled
with an error-feedback mechanism used widely in optimization, exhibit the same
non-asymptotic theoretical guarantees as their SGD counterparts. Second, we
show that our analysis framework extends seamlessly to nonlinear stochastic
approximation schemes that subsume Q-learning. Third, we prove that for
multi-agent TD learning, one can achieve linear convergence speedups with
respect to the number of agents while communicating just $\tilde{O}(1)$ bits
per iteration. Notably, these are the first finite-time results in RL that
account for general compression operators and error-feedback in tandem with
linear function approximation and Markovian sampling. Our proofs hinge on the
construction of novel Lyapunov functions that capture the dynamics of a memory
variable introduced by error-feedback.
