  New malware emerges at a rapid pace and often incorporates Domain Generation
Algorithms (DGAs) to avoid blocking the malware's connection to the command and
control (C2) server. Current state-of-the-art classifiers are able to separate
benign from malicious domains (binary classification) and attribute them with
high probability to the DGAs that generated them (multiclass classification).
While binary classifiers can label domains of yet unknown DGAs as malicious,
multiclass classifiers can only assign domains to DGAs that are known at the
time of training, limiting the ability to uncover new malware families. In this
work, we perform a comprehensive study on the detection of new DGAs, which
includes an evaluation of 59,690 classifiers. We examine four different
approaches in 15 different configurations and propose a simple yet effective
approach based on the combination of a softmax classifier and regular
expressions (regexes) to detect multiple unknown DGAs with high probability. At
the same time, our approach retains state-of-the-art classification performance
for known DGAs. Our evaluation is based on a leave-one-group-out
cross-validation with a total of 94 DGA families. By using the maximum number
of known DGAs, our evaluation scenario is particularly difficult and close to
the real world. All of the approaches examined are privacy-preserving, since
they operate without context and exclusively on a single domain to be
classified. We round up our study with a thorough discussion of
class-incremental learning strategies that can adapt an existing classifier to
newly discovered classes.
