  Privacy policies provide individuals with information about their rights and
how their personal information is handled. Natural language understanding (NLU)
technologies can support individuals and practitioners to understand better
privacy practices described in lengthy and complex documents. However, existing
efforts that use NLU technologies are limited by processing the language in a
way exclusive to a single task focusing on certain privacy practices. To this
end, we introduce the Privacy Policy Language Understanding Evaluation (PLUE)
benchmark, a multi-task benchmark for evaluating the privacy policy language
understanding across various tasks. We also collect a large corpus of privacy
policies to enable privacy policy domain-specific language model pre-training.
We evaluate several generic pre-trained language models and continue
pre-training them on the collected corpus. We demonstrate that domain-specific
continual pre-training offers performance improvements across all tasks.
