  The volume, variety, and velocity of change in vulnerabilities and exploits
have made incident threat analysis challenging with human expertise and
experience along. Tactics, Techniques, and Procedures (TTPs) are to describe
how and why attackers exploit vulnerabilities. However, a TTP description
written by one security professional can be interpreted very differently by
another, leading to confusion in cybersecurity operations or even business,
policy, and legal decisions. Meanwhile, advancements in AI have led to the
increasing use of Natural Language Processing (NLP) algorithms to assist the
various tasks in cyber operations. With the rise of Large Language Models
(LLMs), NLP tasks have significantly improved because of the LLM's semantic
understanding and scalability. This leads us to question how well LLMs can
interpret TTPs or general cyberattack descriptions to inform analysts of the
intended purposes of cyberattacks. We propose to analyze and compare the direct
use of LLMs (e.g., GPT-3.5) versus supervised fine-tuning (SFT) of
small-scale-LLMs (e.g., BERT) to study their capabilities in predicting ATT&CK
tactics. Our results reveal that the small-scale-LLMs with SFT provide a more
focused and clearer differentiation between the ATT&CK tactics (if such
differentiation exists). On the other hand, direct use of LLMs offer a broader
interpretation of cyberattack techniques. When treating more general cases,
despite the power of LLMs, inherent ambiguity exists and limits their
predictive power. We then summarize the challenges and recommend research
directions on LLMs to treat the inherent ambiguity of TTP descriptions used in
various cyber operations.
