  Penetration testing, an essential component of software security testing,
allows organizations to identify and remediate vulnerabilities in their
systems, thus bolstering their defense mechanisms against cyberattacks. One
recent advancement in the realm of penetration testing is the utilization of
Language Models (LLMs). We explore the intersection of LLMs and penetration
testing to gain insight into their capabilities and challenges in the context
of privilege escalation. We introduce a fully automated privilege-escalation
tool designed for evaluating the efficacy of LLMs for (ethical) hacking,
executing benchmarks using multiple LLMs, and investigating their respective
results.
  Our results show that GPT-4-turbo is well suited to exploit vulnerabilities
(33-83% of vulnerabilities). GPT-3.5-turbo can abuse 16-50% of vulnerabilities,
while local models, such as Llama3, can only exploit between 0 and 33% of the
vulnerabilities.
  We analyze the impact of different context sizes, in-context learning,
optional high-level guidance mechanisms, and memory management techniques. We
discuss challenging areas for LLMs, including maintaining focus during testing,
coping with errors, and finally comparing LLMs with human hackers.
  The current version of the LLM-guided privilege-escalation prototype can be
found at https://github.com/ipa-labs/hackingBuddyGPT.
