  Context: One of the black arts of data mining is learning the magic
parameters which control the learners. In software analytics, at least for
defect prediction, several methods, like grid search and differential evolution
(DE), have been proposed to learn these parameters, which has been proved to be
able to improve the performance scores of learners.
  Objective: We want to evaluate which method can find better parameters in
terms of performance score and runtime cost.
  Methods: This paper compares grid search to differential evolution, which is
an evolutionary algorithm that makes extensive use of stochastic jumps around
the search space.
  Results: We find that the seemingly complete approach of grid search does no
better, and sometimes worse, than the stochastic search. When repeated 20 times
to check for conclusion validity, DE was over 210 times faster than grid search
to tune Random Forests on 17 testing data sets with F-Measure
  Conclusions: These results are puzzling: why does a quick partial search be
just as effective as a much slower, and much more, extensive search? To answer
that question, we turned to the theoretical optimization literature. Bergstra
and Bengio conjecture that grid search is not more effective than more
randomized searchers if the underlying search space is inherently low
dimensional. This is significant since recent results show that defect
prediction exhibits very low intrinsic dimensionality-- an observation that
explains why a fast method like DE may work as well as a seemingly more
thorough grid search. This suggests, as a future research direction, that it
might be possible to peek at data sets before doing any optimization in order
to match the optimization algorithm to the problem at hand.
