  Context: Systematic literature reviews (SLRs) rely on a rigorous and
auditable methodology for minimizing biases and ensuring reliability. A common
kind of bias arises when selecting studies using a set of inclusion/exclusion
criteria. This bias can be decreased through dual revision, which makes the
selection process more time-consuming and remains prone to generating bias
depending on how each researcher interprets the inclusion/exclusion criteria.
Objective: To reduce the bias and time spent in the study selection process,
this paper presents a process for selecting studies based on the use of Cohen's
Kappa statistic. We have defined an iterative process based on the use of this
statistic during which the criteria are refined until obtain almost perfect
agreement (k>0.8). At this point, the two researchers interpret the selection
criteria in the same way, and thus, the bias is reduced. Starting from this
agreement, dual review can be eliminated; consequently, the time spent is
drastically shortened. Method: The feasibility of this iterative process for
selecting studies is demonstrated through a tertiary study in the area of
software engineering on works that were published from 2005 to 2018. Results:
The time saved in the study selection process was 28% (for 152 studies) and if
the number of studies is sufficiently large, the time saved tend asymptotically
to 50%. Conclusions: Researchers and students may take advantage of this
iterative process for selecting studies when conducting SLRs to reduce bias in
the interpretation of inclusion and exclusion criteria. It is especially useful
for research with few resources.
