  Almost every software system provides configuration options to tailor the
system to the target platform and application scenario. Often, this
configurability renders the analysis of every individual system configuration
infeasible. To address this problem, researchers have proposed a diverse set of
sampling algorithms. We present a comparative study of 10 state-of-the-art
sampling algorithms regarding their fault-detection capability and size of
sample sets. The former is important to improve software quality and the latter
to reduce the time of analysis. In a nutshell, we found that sampling
algorithms with larger sample sets are able to detect higher numbers of faults,
but simple algorithms with small sample sets, such as most-enabled-disabled,
are the most efficient in most contexts. Furthermore, we observed that the
limiting assumptions made in previous work influence the number of detected
faults, the size of sample sets, and the ranking of algorithms. Finally, we
have identified a number of technical challenges when trying to avoid the
limiting assumptions, which questions the practicality of certain sampling
algorithms.
