  We propose LLMA, an LLM accelerator to losslessly speed up Large Language
Model (LLM) inference with references. LLMA is motivated by the observation
that there are abundant identical text spans between the decoding result by an
LLM and the reference that is available in many real world scenarios (e.g.,
retrieved documents). LLMA first selects a text span from the reference and
copies its tokens to the decoder and then efficiently checks the tokens'
appropriateness as the decoding result in parallel within one decoding step.
The improved computational parallelism allows LLMA to achieve over 2x speed-up
for LLMs with identical generation results as greedy decoding in many practical
generation scenarios where significant overlap between in-context reference and
outputs exists (e.g., search engines and multi-turn conversations).
