  In this paper, we demonstrate the effectiveness of using the theory of
contextual integrity (CI) to annotate and evaluate privacy policy statements.
We perform a case study using CI annotations to compare Facebook's privacy
policy before and after the Cambridge Analytica scandal. The updated Facebook
privacy policy provides additional details about what information is being
transferred, from whom, by whom, to whom, and under what conditions. However,
some privacy statements prescribe an incomprehensibly large number of
information flows by including many CI parameters in single statements. Other
statements result in incomplete information flows due to the use of vague terms
or omitting contextual parameters altogether. We then demonstrate that
crowdsourcing can effectively produce CI annotations of privacy policies at
scale. We test the CI annotation task on 48 excerpts of privacy policies from
17 companies with 141 crowdworkers. The resulting high precision annotations
indicate that crowdsourcing could be used to produce a large corpus of
annotated privacy policies for future research.
