  [Context and motivation] Incompleteness in natural-language requirements is a
challenging problem. [Question/problem] A common technique for detecting
incompleteness in requirements is checking the requirements against external
sources. With the emergence of language models such as BERT, an interesting
question is whether language models are useful external sources for finding
potential incompleteness in requirements. [Principal ideas/results] We mask
words in requirements and have BERT's masked language model (MLM) generate
contextualized predictions for filling the masked slots. We simulate
incompleteness by withholding content from requirements and measure BERT's
ability to predict terminology that is present in the withheld content but
absent in the content disclosed to BERT. [Contribution] BERT can be configured
to generate multiple predictions per mask. Our first contribution is to
determine how many predictions per mask is an optimal trade-off between
effectively discovering omissions in requirements and the level of noise in the
predictions. Our second contribution is devising a machine learning-based
filter that post-processes predictions made by BERT to further reduce noise. We
empirically evaluate our solution over 40 requirements specifications drawn
from the PURE dataset [1]. Our results indicate that: (1) predictions made by
BERT are highly effective at pinpointing terminology that is missing from
requirements, and (2) our filter can substantially reduce noise from the
predictions, thus making BERT a more compelling aid for improving completeness
in requirements.
