  In recent years, deep learning gained proliferating popularity in the
cybersecurity application domain, since when being compared to traditional
machine learning, it usually involves less human effort, produces better
results, and provides better generalizability. However, the imbalanced data
issue is very common in cybersecurity, which can substantially deteriorate the
performance of the deep learning models. This paper introduces a transfer
learning based method to tackle the imbalanced data issue in cybersecurity
using Return-Oriented Programming (ROP) payload detection as a case study. We
achieved 0.033 average false positive rate, 0.9718 average F1 score and 0.9418
average detection rate on 3 different target domain programs using 2 different
source domain programs, with 0 benign training data samples in the target
domain. The performance improvement compared to the baseline is a trade-off
between false positive rate and detection rate. Using our approach, the number
of false positives is reduced by 23.20%, and as a trade-off, the number of
detected malicious samples is reduced by 0.50%.
