  Machine learning (ML) is promising in accurately detecting malicious flows in
encrypted network traffic; however, it is challenging to collect a training
dataset that contains a sufficient amount of encrypted malicious data with
correct labels. When ML models are trained with low-quality training data, they
suffer degraded performance. In this paper, we aim at addressing a real-world
low-quality training dataset problem, namely, detecting encrypted malicious
traffic generated by continuously evolving malware. We develop RAPIER that
fully utilizes different distributions of normal and malicious traffic data in
the feature space, where normal data is tightly distributed in a certain area
and the malicious data is scattered over the entire feature space to augment
training data for model training. RAPIER includes two pre-processing modules to
convert traffic into feature vectors and correct label noises. We evaluate our
system on two public datasets and one combined dataset. With 1000 samples and
45% noises from each dataset, our system achieves the F1 scores of 0.770,
0.776, and 0.855, respectively, achieving average improvements of 352.6%,
284.3%, and 214.9% over the existing methods, respectively. Furthermore, We
evaluate RAPIER with a real-world dataset obtained from a security enterprise.
RAPIER effectively achieves encrypted malicious traffic detection with the best
F1 score of 0.773 and improves the F1 score of existing methods by an average
of 272.5%.
