  Large language models (LLMs) trained on datasets of publicly available source
code have established a new state of the art in code generation tasks. However,
these models are mostly unaware of the code that exists within a specific
project, preventing the models from making good use of existing APIs. Instead,
LLMs often invent, or "hallucinate", non-existent APIs or produce variants of
already existing code. This paper presents De-Hallucinator, a technique that
grounds the predictions of an LLM through a novel combination of retrieving
suitable API references and iteratively querying the model with increasingly
suitable context information in the prompt. The approach exploits the
observation that predictions by LLMs often resemble the desired code, but they
fail to correctly refer to already existing APIs. De-Hallucinator automatically
identifies project-specific API references related to the model's initial
predictions and adds these references into the prompt. Unlike
retrieval-augmented generation (RAG), our approach uses the initial
prediction(s) by the model to iteratively retrieve increasingly suitable API
references. Our evaluation applies the approach to two tasks: predicting API
usages in Python and generating tests in JavaScript. We show that
De-Hallucinator consistently improves the generated code across five LLMs. In
particular, the approach improves the edit distance by 23.3-50.6% and the
recall of correctly predicted API usages by 23.9-61.0% for code completion, and
improves the number of fixed tests that initially failed because of
hallucinations by 63.2%, resulting in a 15.5% increase in statement coverage
for test generation.
