  The $S_8$ tension between low-redshift galaxy surveys and the primary CMB
signals a possible breakdown of the $\Lambda$CDM model. Recently differing
results have been obtained using low-redshift galaxy surveys and the higher
redshifts probed by CMB lensing, motivating a possible time-dependent
modification to the growth of structure. We investigate a simple
phenomenological model in which the growth of structure deviates from the
$\Lambda$CDM prediction at late times, in particular as a simple function of
the dark energy density. Fitting to galaxy lensing, CMB lensing, BAO, and
Supernovae datasets, we find significant evidence - 2.5 - 3$\sigma$, depending
on analysis choices - for a non-zero value of the parameter quantifying a
deviation from $\Lambda$CDM. The preferred model, which has a slower growth of
structure below $z\sim 1$, improves the joint fit to the data over
$\Lambda$CDM. While the overall fit is improved, there is weak evidence for
galaxy and CMB lensing favoring different changes in the growth of structure.
