  The fast advancements in Large Language Models (LLMs) are driving an
increasing number of applications. Together with the growing number of users,
we also see an increasing number of attackers who try to outsmart these
systems. They want the model to reveal confidential information, specific false
information, or offensive behavior. To this end, they manipulate their
instructions for the LLM by inserting separators or rephrasing them
systematically until they reach their goal. Our approach is different. It
inserts words from the model vocabulary. We find these words using an
optimization procedure and embeddings from another LLM (attacker LLM). We prove
our approach by goal hijacking two popular open-source LLMs from the Llama2 and
the Flan-T5 families, respectively. We present two main findings. First, our
approach creates inconspicuous instructions and therefore it is hard to detect.
For many attack cases, we find that even a single word insertion is sufficient.
Second, we demonstrate that we can conduct our attack using a different model
than the target model to conduct our attack with.
