  Dependabot, a popular dependency management tool, includes a compatibility
score feature that helps client packages assess the risk of accepting a
dependency update by leveraging knowledge from "the crowd". For each dependency
update, Dependabot calculates this compatibility score as the proportion of
successful updates performed by other client packages that use the same
provider package as a dependency. In this paper, we study the efficacy of the
compatibility score to help client packages assess the risks involved with
accepting a dependency update. We analyze 579,206 pull requests opened by
Dependabot to update a dependency, along with 618,045 compatibility score
records calculated by Dependabot. We find that a compatibility score cannot be
calculated for 83% of the dependency updates due to the lack of data from the
crowd. Yet, the vast majority of the scores that can be calculated have a small
confidence interval and are based on low-quality data, suggesting that client
packages should have additional angles to evaluate the risk of an update and
the trustworthiness of the compatibility score. To overcome these limitations,
we propose metrics that amplify the input from the crowd and demonstrate the
ability of those metrics to predict the acceptance of a successful update by
client packages. We also demonstrate that historical update metrics from client
packages can be used to provide a more personalized compatibility score. Based
on our findings, we argue that, when leveraging the crowd, dependency
management bots should include a confidence interval to help calibrate the
trust clients can place in the compatibility score, and consider the quality of
tests that exercise candidate updates.
