  Observational data about human behavior is often heterogeneous, i.e.,
generated by subgroups within the population under study that vary in size and
behavior. Heterogeneity predisposes analysis to Simpson's paradox, whereby the
trends observed in data that has been aggregated over the entire population may
be substantially different from those of the underlying subgroups. I illustrate
Simpson's paradox with several examples coming from studies of online behavior
and show that aggregate response leads to wrong conclusions about the
underlying individual behavior. I then present a simple method to test whether
Simpson's paradox is affecting results of analysis. The presence of Simpson's
paradox in social data suggests that important behavioral differences exist
within the population, and failure to take these differences into account can
distort the studies' findings.
