  We examine the dynamics of Rydberg polaritons with dipolar interactions that
propagate in multiple spatial modes. The dipolar excitation exchange between
different Rydberg states mediates an effective exchange between polaritons that
enables photons to hop across different spatial channels. Remarkably, the
efficiency of this photon exchange process can increase with the channel
distance and becomes optimal at a finite rail separation. Based on this
mechanism, we design a simple photonic network that realizes a two photon
quantum gate with a robust $\pi$-phase, protected by the symmetries of the
underlying photon interaction and the geometry of the network. These
capabilities expand the scope of Rydberg electromagnetically induced
transparency towards multidimensional geometries for nonlinear optical networks
and explorations of photonic many-body physics.
