  There is an increasing trend to mine vulnerabilities from software
repositories and use machine learning techniques to automatically detect
software vulnerabilities. A fundamental but unresolved research question is:
how do different factors in the mining and learning process impact the accuracy
of identifying vulnerabilities in software projects of varying characteristics?
Substantial research has been dedicated in this area, including source code
static analysis, software repository mining, and NLP-based machine learning.
However, practitioners lack experience regarding the key factors for building a
baseline model of the state-of-the-art. In addition, there lacks of experience
regarding the transferability of the vulnerability signatures from project to
project. This study investigates how the combination of different vulnerability
features and three representative machine learning models impact the accuracy
of vulnerability detection in 17 real-world projects. We examine two types of
vulnerability representations: 1) code features extracted through NLP with
varying tokenization strategies and three different embedding techniques
(bag-of-words, word2vec, and fastText) and 2) a set of eight architectural
metrics that capture the abstract design of the software systems. The three
machine learning algorithms include a random forest model, a support vector
machines model, and a residual neural network model. The analysis shows a
recommended baseline model with signatures extracted through bag-of-words
embedding, combined with the random forest, consistently increases the
detection accuracy by about 4% compared to other combinations in all 17
projects. Furthermore, we observe the limitation of transferring vulnerability
signatures across domains based on our experiments.
