  Recent work raises concerns about the use of standard splits to compare
natural language processing models. We propose a Bayesian statistical model
comparison technique which uses k-fold cross-validation across multiple data
sets to estimate the likelihood that one model will outperform the other, or
that the two will produce practically equivalent results. We use this technique
to rank six English part-of-speech taggers across two data sets and three
evaluation metrics.
