  The cross-prompt injection attack (XPIA) is an effective technique that can
be used for data exfiltration, and that has seen increasing use. In this
attack, the attacker injects a malicious instruction into third party data
which an LLM is likely to consume when assisting a user, who is the victim.
XPIA is often used as a means for data exfiltration, and the estimated cost of
the average data breach for a business is nearly $4.5 million, which includes
breaches such as compromised enterprise credentials. With the rise of
gradient-based attacks such as the GCG suffix attack, the odds of an XPIA
occurring which uses a GCG suffix are worryingly high. As part of my work in
Microsoft's AI Red Team, I demonstrated a viable attack model using a GCG
suffix paired with an injection in a simulated XPIA scenario. The results
indicate that the presence of a GCG suffix can increase the odds of successful
data exfiltration by nearly 20%, with some caveats.
