  Privacy policies of websites are often lengthy and intricate. Privacy
assistants assist in simplifying policies and making them more accessible and
user friendly. The emergence of generative AI (genAI) offers new opportunities
to build privacy assistants that can answer users questions about privacy
policies. However, genAIs reliability is a concern due to its potential for
producing inaccurate information. This study introduces GenAIPABench, a
benchmark for evaluating Generative AI-based Privacy Assistants (GenAIPAs).
GenAIPABench includes: 1) A set of questions about privacy policies and data
protection regulations, with annotated answers for various organizations and
regulations; 2) Metrics to assess the accuracy, relevance, and consistency of
responses; and 3) A tool for generating prompts to introduce privacy documents
and varied privacy questions to test system robustness. We evaluated three
leading genAI systems ChatGPT-4, Bard, and Bing AI using GenAIPABench to gauge
their effectiveness as GenAIPAs. Our results demonstrate significant promise in
genAI capabilities in the privacy domain while also highlighting challenges in
managing complex queries, ensuring consistency, and verifying source accuracy.
