  Automatic citation generation for sentences in a document or report is
paramount for intelligence analysts, cybersecurity, news agencies, and
education personnel. In this research, we investigate whether large language
models (LLMs) are capable of generating references based on two forms of
sentence queries: (a) Direct Queries, LLMs are asked to provide author names of
the given research article, and (b) Indirect Queries, LLMs are asked to provide
the title of a mentioned article when given a sentence from a different
article. To demonstrate where LLM stands in this task, we introduce a large
dataset called REASONS comprising abstracts of the 12 most popular domains of
scientific research on arXiv. From around 20K research articles, we make the
following deductions on public and proprietary LLMs: (a) State-of-the-art,
often called anthropomorphic GPT-4 and GPT-3.5, suffers from high pass
percentage (PP) to minimize the hallucination rate (HR). When tested with
Perplexity.ai (7B), they unexpectedly made more errors; (b) Augmenting relevant
metadata lowered the PP and gave the lowest HR; (c) Advance retrieval-augmented
generation (RAG) using Mistral demonstrates consistent and robust citation
support on indirect queries and matched performance to GPT-3.5 and GPT-4. The
HR across all domains and models decreased by an average of 41.93%, and the PP
was reduced to 0% in most cases. In terms of generation quality, the average F1
Score and BLEU were 68.09% and 57.51%, respectively; (d) Testing with
adversarial samples showed that LLMs, including the Advance RAG Mistral,
struggle to understand context, but the extent of this issue was small in
Mistral and GPT-4-Preview. Our study contributes valuable insights into the
reliability of RAG for automated citation generation tasks.
