  Change-based testing is a key component of continuous integration at
Facebook. However, a large number of tests coupled with a high rate of changes
committed to our monolithic repository make it infeasible to run all
potentially-impacted tests on each change. We propose a new predictive test
selection strategy which selects a subset of tests to exercise for each change
submitted to the continuous integration system. The strategy is learned from a
large dataset of historical test outcomes using basic machine learning
techniques. Deployed in production, the strategy reduces the total
infrastructure cost of testing code changes by a factor of two, while
guaranteeing that over 95% of individual test failures and over 99.9% of faulty
changes are still reported back to developers. The method we present here also
accounts for the non-determinism of test outcomes, also known as test
flakiness.
