  In literature, we can find research tools to automatically generate test
cases for RESTful APIs, addressing the specificity of this particular
programming domain. However, no direct comparison of these tools is available
to guide developers in deciding which tool best fits their REST API project. In
this paper, we present the results of an empirical comparison of automated
black-box test case generation approaches for REST APIs. We surveyed the
available black-box testing tools that have been proposed in recent literature,
finding four usable prototypes: RestTestGen, RESTler, bBOXRT and RESTest. We
used these tools to generate test cases for 14 real-world REST services. Then,
testing results have been analyzed and compared in terms of robustness (i.e.,
success rate) and test coverage. Among the considered tools, RESTler appears to
be the most solid, able to successfully test all case studies (the other tools
experienced crashes). Conversely, test cases generated by RestTestGen scored
the highest coverage, suggesting that its testing strategy is the most
effective in testing REST APIs.
