  Cyber threat intelligence (CTI) is crucial in today's cybersecurity
landscape, providing essential insights to understand and mitigate the
ever-evolving cyber threats. The recent rise of Large Language Models (LLMs)
have shown potential in this domain, but concerns about their reliability,
accuracy, and hallucinations persist. While existing benchmarks provide general
evaluations of LLMs, there are no benchmarks that address the practical and
applied aspects of CTI-specific tasks. To bridge this gap, we introduce
CTIBench, a benchmark designed to assess LLMs' performance in CTI applications.
CTIBench includes multiple datasets focused on evaluating knowledge acquired by
LLMs in the cyber-threat landscape. Our evaluation of several state-of-the-art
models on these tasks provides insights into their strengths and weaknesses in
CTI contexts, contributing to a better understanding of LLM capabilities in
CTI.
