  Crowdsourcing and data mining can be used to effectively reduce the effort
associated with the partial replication and enhancement of qualitative studies.
  For example, in a primary study, other researchers explored factors
influencing the fate of GitHub pull requests using an extensive qualitative
analysis of 20 pull requests. Guided by their findings, we mapped some of their
qualitative insights onto quantitative questions. To determine how well their
findings generalize, we collected much more data (170 additional pull requests
from 142 GitHub projects). Using crowdsourcing, that data was augmented with
subjective qualitative human opinions about how pull requests extended the
original issue. The crowd's answers were then combined with quantitative
features and, using data mining, used to build a predictor for whether code
would be merged. That predictor was far more accurate that one built from the
primary study's qualitative factors (F1=90 vs 68\%), illustrating the value of
a mixed-methods approach and replication to improve prior results.
  To test the generality of this approach, the next step in future work is to
conduct other studies that extend qualitative studies with crowdsourcing and
data mining.
