  In training their newly-developed malware detection methods, researchers rely
on threshold-based labeling strategies that interpret the scan reports provided
by online platforms, such as VirusTotal. The dynamicity of this platform
renders those labeling strategies unsustainable over prolonged periods, which
leads to inaccurate labels. Using inaccurately labeled apps to train and
evaluate malware detection methods significantly undermines the reliability of
their results, leading to either dismissing otherwise promising detection
approaches or adopting intrinsically inadequate ones. The infeasibility of
generating accurate labels via manual analysis and the lack of reliable
alternatives force researchers to utilize VirusTotal to label apps. In the
paper, we tackle this issue in two manners. Firstly, we reveal the aspects of
VirusTotal's dynamicity and how they impact threshold-based labeling strategies
and provide actionable insights on how to use these labeling strategies given
VirusTotal's dynamicity reliably. Secondly, we motivate the implementation of
alternative platforms by (a) identifying VirusTotal limitations that such
platforms should avoid, and (b) proposing an architecture of how such platforms
can be constructed to mitigate VirusTotal's limitations.
