  Newberry et al. (Detecting evolutionary forces in language change, Nature
551, 2017) tackle an important but difficult problem in linguistics, the
testing of selective theories of language change against a null model of drift.
Having applied a test from population genetics (the Frequency Increment Test)
to a number of relevant examples, they suggest stochasticity has a previously
under-appreciated role in language evolution. We replicate their results and
find that while the overall observation holds, results produced by this
approach on individual time series can be sensitive to how the corpus is
organized into temporal segments (binning). Furthermore, we use a large set of
simulations in conjunction with binning to systematically explore the range of
applicability of the Frequency Increment Test. We conclude that care should be
exercised with interpreting results of tests like the Frequency Increment Test
on individual series, given the researcher degrees of freedom available when
applying the test to corpus data, and fundamental differences between genetic
and linguistic data. Our findings have implications for selection testing and
temporal binning in general, as well as demonstrating the usefulness of
simulations for evaluating methods newly introduced to the field.
