  We provide a new non-asymptotic analysis of distributed temporal difference
learning with linear function approximation. Our approach relies on ``one-shot
averaging,'' where $N$ agents run identical local copies of the TD(0) method
and average the outcomes only once at the very end. We demonstrate a version of
the linear time speedup phenomenon, where the convergence time of the
distributed process is a factor of $N$ faster than the convergence time of
TD(0). This is the first result proving benefits from parallelism for temporal
difference methods.
