  Universal Information Extraction~(Universal IE) aims to solve different
extraction tasks in a uniform text-to-structure generation manner. Such a
generation procedure tends to struggle when there exist complex information
structures to be extracted. Retrieving knowledge from external knowledge bases
may help models to overcome this problem but it is impossible to construct a
knowledge base suitable for various IE tasks. Inspired by the fact that large
amount of knowledge are stored in the pretrained language models~(PLM) and can
be retrieved explicitly, in this paper, we propose MetaRetriever to retrieve
task-specific knowledge from PLMs to enhance universal IE. As different IE
tasks need different knowledge, we further propose a Meta-Pretraining Algorithm
which allows MetaRetriever to quicktly achieve maximum task-specific retrieval
performance when fine-tuning on downstream IE tasks. Experimental results show
that MetaRetriever achieves the new state-of-the-art on 4 IE tasks, 12 datasets
under fully-supervised, low-resource and few-shot scenarios.
