  Change point detection has recently gained popularity as a method of
detecting performance changes in software due to its ability to cope with noisy
data. In this paper we present Hunter, an open source tool that automatically
detects performance regressions and improvements in time-series data. Hunter
uses a modified E-divisive means algorithm to identify statistically
significant changes in normally-distributed performance metrics. We describe
the changes we made to the E-divisive means algorithm along with their
motivation. The main change we adopted was to replace the significance test
using randomized permutations with a Student's t-test, as we discovered that
the randomized approach did not produce deterministic results, at least not
with a reasonable number of iterations. In addition we've made tweaks that
allow us to find change points the original algorithm would not, such as two
nearby changes. For evaluation, we developed a method to generate real
timeseries, but with artificially injected changes in latency. We used these
data sets to compare Hunter against two other well known algorithms, PELT and
DYNP. Finally, we conclude with lessons we've learned supporting Hunter across
teams with individual responsibility for the performance of their project.
