  Understanding privacy policies is crucial for users as it empowers them to
learn about the information that matters to them. Sentences written in a
privacy policy document explain privacy practices, and the constituent text
spans convey further specific information about that practice. We refer to
predicting the privacy practice explained in a sentence as intent
classification and identifying the text spans sharing specific information as
slot filling. In this work, we propose PolicyIE, an English corpus consisting
of 5,250 intent and 11,788 slot annotations spanning 31 privacy policies of
websites and mobile applications. PolicyIE corpus is a challenging real-world
benchmark with limited labeled examples reflecting the cost of collecting
large-scale annotations from domain experts. We present two alternative neural
approaches as baselines, (1) intent classification and slot filling as a joint
sequence tagging and (2) modeling them as a sequence-to-sequence (Seq2Seq)
learning task. The experiment results show that both approaches perform
comparably in intent classification, while the Seq2Seq method outperforms the
sequence tagging approach in slot filling by a large margin. We perform a
detailed error analysis to reveal the challenges of the proposed corpus.
