  Web API specifications are machine-readable descriptions of APIs. These
specifications, in combination with related tooling, simplify and support the
consumption of APIs. However, despite the increased distribution of web APIs,
specifications are rare and their creation and maintenance heavily relies on
manual efforts by third parties. In this paper, we propose an automatic
approach and an associated tool called D2Spec for extracting specifications
from web API documentation pages. Given a seed online documentation page on an
API, D2Spec first crawls all documentation pages on the API, and then uses a
set of machine learning techniques to extract the base URL, path templates, and
HTTP methods, which collectively describe the endpoints of an API. We evaluated
whether D2Spec can accurately extract endpoints from documentation on 120 web
APIs. The results showed that D2Spec achieved a precision of 87.5% in
identifying base URLs, a precision of 81.3% and a recall of 80.6% in generating
path templates, and a precision of 84.4% and a recall of 76.2% in extracting
HTTP methods. In addition, we found that D2Spec was useful when applied to APIs
with pre-existing API specifications: D2Spec revealed many inconsistencies
between web API documentation and their corresponding publicly available
specifications. Thus, D2Spec can be used by web API providers to keep
documentation and specifications in synchronization.
