  Modern web services routinely provide REST APIs for clients to access their
functionality. These APIs present unique challenges and opportunities for
automated testing, driving the recent development of many techniques and tools
that generate test cases for API endpoints using various strategies.
Understanding how these techniques compare to one another is difficult, as they
have been evaluated on different benchmarks and using different metrics. To
fill this gap, we performed an empirical study aimed to understand the
landscape in automated testing of REST APIs and guide future research in this
area. We first identified, through a systematic selection process, a set of 10
state-of-the-art REST API testing tools that included tools developed by both
researchers and practitioners. We then applied these tools to a benchmark of 20
real-world open-source RESTful services and analyzed their performance in terms
of code coverage achieved and unique failures triggered. This analysis allowed
us to identify strengths, weaknesses, and limitations of the tools considered
and of their underlying strategies, as well as implications of our findings for
future research in this area.
