  Application Programming Interfaces (APIs), which encapsulate the
implementation of specific functions as interfaces, greatly improve the
efficiency of modern software development. As numbers of APIs spring up
nowadays, developers can hardly be familiar with all the APIs, and usually need
to search for appropriate APIs for usage. So lots of efforts have been devoted
to improving the API recommendation task. However, it has been increasingly
difficult to gauge the performance of new models due to the lack of a uniform
definition of the task and a standardized benchmark. For example, some studies
regard the task as a code completion problem; while others recommend relative
APIs given natural language queries. To reduce the challenges and better
facilitate future research, in this paper, we revisit the API recommendation
task and aim at benchmarking the approaches. Specifically, the paper groups the
approaches into two categories according to the task definition, i.e.,
query-based API recommendation and code-based API recommendation. We study 11
recently-proposed approaches along with 4 widely-used IDEs. One benchmark named
as APIBench is then built for the two respective categories of approaches.
Based on APIBench, we distill some actionable insights and challenges for API
recommendation. We also achieve some implications and directions for improving
the performance of recommending APIs, including data source selection,
appropriate query reformulation, low resource setting, and cross-domain
adaptation.
