  Large multimodal language models have proven transformative in numerous
applications. However, these models have been shown to memorize and leak
pre-training data, raising serious user privacy and information security
concerns. While data leaks should be prevented, it is also crucial to examine
the trade-off between the privacy protection and model utility of proposed
approaches. In this paper, we introduce PrivQA -- a multimodal benchmark to
assess this privacy/utility trade-off when a model is instructed to protect
specific categories of personal information in a simulated scenario. We also
propose a technique to iteratively self-moderate responses, which significantly
improves privacy. However, through a series of red-teaming experiments, we find
that adversaries can also easily circumvent these protections with simple
jailbreaking methods through textual and/or image inputs. We believe PrivQA has
the potential to support the development of new models with improved privacy
protections, as well as the adversarial robustness of these protections. We
release the entire PrivQA dataset at https://llm-access-control.github.io/.
