  Pretraining auto-regressive large language models~(LLMs) with retrieval
demonstrates better perplexity and factual accuracy by leveraging external
databases. However, the size of existing pretrained retrieval-augmented LLM is
still limited (e.g., Retro has 7.5B parameters), which limits the effectiveness
of instruction tuning and zero-shot generalization. In this work, we introduce
Retro 48B, the largest LLM pretrained with retrieval. Specifically, we continue
to pretrain a 43B GPT model on additional 100 billion tokens using the Retro
augmentation method by retrieving from 1.2 trillion tokens. Notably, the
obtained foundation model, Retro 48B, largely outperforms the counterpart GPT
43B trained on 1.2T tokens in terms of perplexity with only 2.58% additional
GPU hours, demonstrating the significant scaling potential of the method. After
instruction tuning on Retro, InstructRetro demonstrates significant improvement
over the instruction tuned GPT on a wide range of zero-shot tasks.
Specifically, the average improvement of InstructRetro is 7% over its GPT
counterpart across 8 short-form QA and reading comprehension tasks, 10% over
GPT across 4 challenging long-form QA tasks, and 16% over GPT across 3
summarization tasks. Surprisingly, we find that one can ablate the encoder from
InstructRetro architecture and directly use its decoder backbone, while
achieving comparable results. Our results highlight the promising direction to
obtain a better GPT decoder through continued pretraining with retrieval before
instruction tuning. Our code and checkpoints are publicly available at:
https://huggingface.co/nvidia/retro-48b-instruct-4k.
