  We experimentally study the effects of the anisotropic Rydberg-interaction on
$D$-state Rydberg polaritons slowly propagating through a cold atomic sample.
In addition to the few-photon nonlinearity known from similar experiments with
Rydberg $S$-states, we observe the interaction-induced dephasing of Rydberg
polaritons at very low photon input rates into the medium. We develop a model
combining the propagation of the two-photon wavefunction through our system
with nonperturbative calculations of the anisotropic Rydberg-interaction to
show that the observed effect can be attributed to pairwise interaction of
individual Rydberg polaritons.
