  Causal inference in observational studies is notoriously difficult, due to
the fact that the experimenter is not in charge of the treatment assignment
mechanism. Many potential con- founding factors (PCFs) exist in such a
scenario, and if one seeks to estimate the causal effect of the treatment on a
response, one needs to control for such factors. Identifying all relevant PCFs
may be difficult (or impossible) given a single observational study. Instead,
we argue that if one can observe a sequence of similar treatments over the
course of a lengthy time period, one can identify patterns of behavior in the
experimental subjects that are correlated with the response of interest and
control for those patterns directly. Specifically, in our case-study we find
and control for an early-adopter effect: the scenario in which the magnitude of
the response is highly correlated with how quickly one adopts a treatment after
its release.
  We provide a flexible hierarchical Bayesian framework that controls for such
early-adopter effects in the analysis of the effects of multiple sequential
treatments. The methods are presented and evaluated in the context of a
detailed case-study involving product updates (newer versions of the same
product) from eBay, Inc. The users in our study upgrade (or not) to a new
version of the product at their own volition and timing. Our response variable
is a measure of user actions, and we study the behavior of a large set of users
(n = 10.5 million) in a targeted subset of eBay categories over a period of one
year. We find that (a) naive causal estimates are hugely misleading and (b) our
method, which is relatively insensitive to modeling assumptions and exhibits
good out-of-sample predictive validation, yields sensible causal estimates that
offer eBay a stable basis for decision-making.
