  With the proliferation of LLM-integrated applications such as GPT-s, millions
are deployed, offering valuable services through proprietary instruction
prompts. These systems, however, are prone to prompt extraction attacks through
meticulously designed queries. To help mitigate this problem, we introduce the
Raccoon benchmark which comprehensively evaluates a model's susceptibility to
prompt extraction attacks. Our novel evaluation method assesses models under
both defenseless and defended scenarios, employing a dual approach to evaluate
the effectiveness of existing defenses and the resilience of the models. The
benchmark encompasses 14 categories of prompt extraction attacks, with
additional compounded attacks that closely mimic the strategies of potential
attackers, alongside a diverse collection of defense templates. This array is,
to our knowledge, the most extensive compilation of prompt theft attacks and
defense mechanisms to date. Our findings highlight universal susceptibility to
prompt theft in the absence of defenses, with OpenAI models demonstrating
notable resilience when protected. This paper aims to establish a more
systematic benchmark for assessing LLM robustness against prompt extraction
attacks, offering insights into their causes and potential countermeasures.
Resources of Raccoon are publicly available at
https://github.com/M0gician/RaccoonBench.
