  This study sheds light on the imperative need to bolster safety and privacy
measures in large language models (LLMs), such as GPT-4 and LLaMA-2, by
identifying and mitigating their vulnerabilities through explainable analysis
of prompt attacks. We propose Counterfactual Explainable Incremental Prompt
Attack (CEIPA), a novel technique where we guide prompts in a specific manner
to quantitatively measure attack effectiveness and explore the embedded defense
mechanisms in these models. Our approach is distinctive for its capacity to
elucidate the reasons behind the generation of harmful responses by LLMs
through an incremental counterfactual methodology. By organizing the prompt
modification process into four incremental levels: (word, sentence, character,
and a combination of character and word) we facilitate a thorough examination
of the susceptibilities inherent to LLMs. The findings from our study not only
provide counterfactual explanation insight but also demonstrate that our
framework significantly enhances the effectiveness of attack prompts.
