  Phishing, whether through email, SMS, or malicious websites, poses a major
threat to organizations by using social engineering to trick users into
revealing sensitive information. It not only compromises company's data
security but also incurs significant financial losses. In this paper, we
investigate whether the remarkable performance of Large Language Models (LLMs)
can be leveraged for particular task like text classification, particularly
detecting malicious content and compare its results with state-of-the-art
Deberta V3 (DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled
Embedding Sharing) model. We systematically assess the potential and
limitations of both approaches using comprehensive public datasets comprising
diverse data sources such as email, HTML, URL, SMS, and synthetic data
generation. Additionally, we demonstrate how LLMs can generate convincing
phishing emails, making it harder to spot scams and evaluate the performance of
both models in this context. Our study delves further into the challenges
encountered by DeBERTa V3 during its training phases, fine-tuning methodology
and transfer learning processes. Similarly, we examine the challenges
associated with LLMs and assess their respective performance. Among our
experimental approaches, the transformer-based DeBERTa method emerged as the
most effective, achieving a test dataset (HuggingFace phishing dataset) recall
(sensitivity) of 95.17% closely followed by GPT-4 providing a recall of 91.04%.
We performed additional experiments with other datasets on the trained DeBERTa
V3 model and LLMs like GPT 4 and Gemini 1.5. Based on our findings, we provide
valuable insights into the effectiveness and robustness of these advanced
language models, offering a detailed comparative analysis that can inform
future research efforts in strengthening cybersecurity measures for detecting
and mitigating phishing threats.
