  We describe our process for automatic detection of performance changes for a
software product in the presence of noise. A large collection of tests run
periodically as changes to our software product are committed to our source
repository, and we would like to identify the commits responsible for
performance regressions. Previously, we relied on manual inspection of time
series graphs to identify significant changes. That was later replaced with a
threshold-based detection system, but neither system was sufficient for finding
changes in performance in a timely manner. This work describes our recent
implementation of a change point detection system built upon the E-Divisive
means algorithm. The algorithm produces a list of change points representing
significant changes from a given history of performance results. A human
reviews the list of change points for actionable changes, which are then
triaged for further inspection. Using change point detection has had a dramatic
impact on our ability to detect performance changes. Quantitatively, it has
dramatically dropped our false positive rate for performance changes, while
qualitatively it has made the entire performance evaluation process easier,
more productive (ex. catching smaller regressions), and more timely.
