  Developers are increasingly using services such as Dependabot to automate
dependency updates. However, recent research has shown that developers perceive
such services as unreliable, as they heavily rely on test coverage to detect
conflicts in updates. To understand the prevalence of tests exercising
dependencies, we calculate the test coverage of direct and indirect uses of
dependencies in 521 well-tested Java projects. We find that tests only cover
58% of direct and 20% of transitive dependency calls. By creating 1,122,420
artificial updates with simple faults covering all dependency usages in 262
projects, we measure the effectiveness of test suites in detecting semantic
faults in dependencies; we find that tests can only detect 47% of direct and
35% of indirect artificial faults on average. To increase reliability, we
investigate the use of change impact analysis as a means of reducing false
negatives; on average, our tool can uncover 74% of injected faults in direct
dependencies and 64% for transitive dependencies, nearly two times more than
test suites. We then apply our tool in 22 real-world dependency updates, where
it identifies three semantically conflicting cases and five cases of unused
dependencies. Our findings indicate that the combination of static and dynamic
analysis should be a requirement for future dependency updating systems.
