  We measure the cross-correlation between the galaxy density in the Dark
Energy Survey (DES) Science Verification data and the lensing of the cosmic
microwave background (CMB) as reconstructed with the Planck satellite and the
South Pole Telescope (SPT). When using the DES main galaxy sample over the full
redshift range $0.2 < z < 1.2$, a cross-correlation signal is detected at $6
\sigma$ and $4\sigma$ with SPT and Planck respectively. We then divide the DES
galaxies into five photometric redshift bins, finding significant ($>$$2
\sigma$) detections in all bins. Comparing to the fiducial Planck cosmology, we
find the redshift evolution of the signal matches expectations, although the
amplitude is consistently lower than predicted across redshift bins. We test
for possible systematics that could affect our result and find no evidence for
significant contamination. Finally, we demonstrate how these measurements can
be used to constrain the growth of structure across cosmic time. We find the
data are fit by a model in which the amplitude of structure in the $z<1.2$
universe is $0.73 \pm 0.16$ times as large as predicted in the LCDM Planck
cosmology, a $1.7\sigma$ deviation.
