  Artificial Intelligence (AI) advancements have enabled the development of
Large Language Models (LLMs) that can perform a variety of tasks with
remarkable semantic understanding and accuracy. ChatGPT is one such LLM that
has gained significant attention due to its impressive capabilities for
assisting in various knowledge-intensive tasks. Due to the knowledge-intensive
nature of engineering secure software, ChatGPT's assistance is expected to be
explored for security-related tasks during the development/evolution of
software. To gain an understanding of the potential of ChatGPT as an emerging
technology for supporting software security, we adopted a two-fold approach.
Initially, we performed an empirical study to analyse the perceptions of those
who had explored the use of ChatGPT for security tasks and shared their views
on Twitter. It was determined that security practitioners view ChatGPT as
beneficial for various software security tasks, including vulnerability
detection, information retrieval, and penetration testing. Secondly, we
designed an experiment aimed at investigating the practicality of this
technology when deployed as an oracle in real-world settings. In particular, we
focused on vulnerability detection and qualitatively examined ChatGPT outputs
for given prompts within this prominent software security task. Based on our
analysis, responses from ChatGPT in this task are largely filled with generic
security information and may not be appropriate for industry use. To prevent
data leakage, we performed this analysis on a vulnerability dataset compiled
after the OpenAI data cut-off date from real-world projects covering 40
distinct vulnerability types and 12 programming languages. We assert that the
findings from this study would contribute to future research aimed at
developing and evaluating LLMs dedicated to software security.
