  Large Language Models (LLMs) have demonstrated potential in cybersecurity
applications but have also caused lower confidence due to problems like
hallucinations and a lack of truthfulness. Existing benchmarks provide general
evaluations but do not sufficiently address the practical and applied aspects
of LLM performance in cybersecurity-specific tasks. To address this gap, we
introduce the SECURE (Security Extraction, Understanding \& Reasoning
Evaluation), a benchmark designed to assess LLMs performance in realistic
cybersecurity scenarios. SECURE includes six datasets focussed on the
Industrial Control System sector to evaluate knowledge extraction,
understanding, and reasoning based on industry-standard sources. Our study
evaluates seven state-of-the-art models on these tasks, providing insights into
their strengths and weaknesses in cybersecurity contexts, and offer
recommendations for improving LLMs reliability as cyber advisory tools.
