  Machine learning (ML) plays a pivotal role in detecting malicious software.
Despite the high F1-scores reported in numerous studies reaching upwards of
0.99, the issue is not completely solved. Malware detectors often experience
performance decay due to constantly evolving operating systems and attack
methods, which can render previously learned knowledge insufficient for
accurate decision-making on new inputs. This paper argues that commonly
reported results are inflated due to two pervasive sources of experimental bias
in the detection task: spatial bias caused by data distributions that are not
representative of a real-world deployment; and temporal bias caused by
incorrect time splits of data, leading to unrealistic configurations. To
address these biases, we introduce a set of constraints for fair experiment
design, and propose a new metric, AUT, for classifier robustness in real-world
settings. We additionally propose an algorithm designed to tune training data
to enhance classifier performance. Finally, we present TESSERACT, an
open-source framework for realistic classifier comparison. Our evaluation
encompasses both traditional ML and deep learning methods, examining published
works on an extensive Android dataset with 259,230 samples over a five-year
span. Additionally, we conduct case studies in the Windows PE and PDF domains.
Our findings identify the existence of biases in previous studies and reveal
that significant performance enhancements are possible through appropriate,
periodic tuning. We explore how mitigation strategies may support in achieving
a more stable and better performance over time by employing multiple strategies
to delay performance decay.
