  Researchers have proposed kinds of malware detection methods to solve the
explosive mobile security threats. We argue that the experiment results are
inflated due to the research bias introduced by the variability of malware
dataset. We explore the impact of bias in Android malware detection in three
aspects, the method used to flag the ground truth, the distribution of malware
families in the dataset, and the methods to use the dataset. We implement a set
of experiments of different VT thresholds and find that the methods used to
flag the malware data affect the malware detection performance directly. We
further compare the impact of malware family types and composition on malware
detection in detail. The superiority of each approach is different under
various combinations of malware families. Through our extensive experiments, we
showed that the methods to use the dataset can have a misleading impact on
evaluation, and the performance difference can be up to over 40%. We argue that
these research biases observed in this paper should be carefully
controlled/eliminated to enforce a fair comparison of malware detection
techniques. Providing reasonable and explainable results is better than only
reporting a high detection accuracy with vague dataset and experimental
settings.
