  To partly address people's concerns over web tracking, Google has created the
Ad Settings webpage to provide information about and some choice over the
profiles Google creates on users. We present AdFisher, an automated tool that
explores how user behaviors, Google's ads, and Ad Settings interact. AdFisher
can run browser-based experiments and analyze data using machine learning and
significance tests. Our tool uses a rigorous experimental design and
statistical analysis to ensure the statistical soundness of our results. We use
AdFisher to find that the Ad Settings was opaque about some features of a
user's profile, that it does provide some choice on ads, and that these choices
can lead to seemingly discriminatory ads. In particular, we found that visiting
webpages associated with substance abuse changed the ads shown but not the
settings page. We also found that setting the gender to female resulted in
getting fewer instances of an ad related to high paying jobs than setting it to
male. We cannot determine who caused these findings due to our limited
visibility into the ad ecosystem, which includes Google, advertisers, websites,
and users. Nevertheless, these results can form the starting point for deeper
investigations by either the companies themselves or by regulatory bodies.
