  Large-scale natural language inference (NLI) datasets such as SNLI or MNLI
have been created by asking crowdworkers to read a premise and write three new
hypotheses, one for each possible semantic relationships (entailment,
contradiction, and neutral). While this protocol has been used to create useful
benchmark data, it remains unclear whether the writing-based annotation
protocol is optimal for any purpose, since it has not been evaluated directly.
Furthermore, there is ample evidence that crowdworker writing can introduce
artifacts in the data. We investigate two alternative protocols which
automatically create candidate (premise, hypothesis) pairs for annotators to
label. Using these protocols and a writing-based baseline, we collect several
new English NLI datasets of over 3k examples each, each using a fixed amount of
annotator time, but a varying number of examples to fit that time budget. Our
experiments on NLI and transfer learning show negative results: None of the
alternative protocols outperforms the baseline in evaluations of generalization
within NLI or on transfer to outside target tasks. We conclude that crowdworker
writing still the best known option for entailment data, highlighting the need
for further data collection work to focus on improving writing-based annotation
processes.
