  In online platforms, the impact of a treatment on an observed outcome may
change over time as 1) users learn about the intervention, and 2) the system
personalization, such as individualized recommendations, change over time. We
introduce a non-parametric causal model of user actions in a personalized
system. We show that the Cookie-Cookie-Day (CCD) experiment, designed for the
measurement of the user learning effect, is biased when there is
personalization. We derive new experimental designs that intervene in the
personalization system to generate the variation necessary to separately
identify the causal effect mediated through user learning and personalization.
Making parametric assumptions allows for the estimation of long-term causal
effects based on medium-term experiments. In simulations, we show that our new
designs successfully recover the dynamic causal effects of interest.
