  As large and powerful neural language models are developed, researchers have
been increasingly interested in developing diagnostic tools to probe them.
There are many papers with conclusions of the form "observation X is found in
model Y", using their own datasets with varying sizes. Larger probing datasets
bring more reliability, but are also expensive to collect. There is yet to be a
quantitative method for estimating reasonable probing dataset sizes. We tackle
this omission in the context of comparing two probing configurations: after we
have collected a small dataset from a pilot study, how many additional data
samples are sufficient to distinguish two different configurations? We present
a novel method to estimate the required number of data samples in such
experiments and, across several case studies, we verify that our estimations
have sufficient statistical power. Our framework helps to systematically
construct probing datasets to diagnose neural NLP models.
