  This open access book provides a comprehensive overview of the state of the
art in research and applications of Foundation Models and is intended for
readers familiar with basic Natural Language Processing (NLP) concepts. Over
the recent years, a revolutionary new paradigm has been developed for training
models for NLP. These models are first pre-trained on large collections of text
documents to acquire general syntactic knowledge and semantic information.
Then, they are fine-tuned for specific tasks, which they can often solve with
superhuman accuracy. When the models are large enough, they can be instructed
by prompts to solve new tasks without any fine-tuning. Moreover, they can be
applied to a wide range of different media and problem domains, ranging from
image and video processing to robot control learning. Because they provide a
blueprint for solving many tasks in artificial intelligence, they have been
called Foundation Models. After a brief introduction to basic NLP models the
main pre-trained language models BERT, GPT and sequence-to-sequence transformer
are described, as well as the concepts of self-attention and context-sensitive
embedding. Then, different approaches to improving these models are discussed,
such as expanding the pre-training criteria, increasing the length of input
texts, or including extra knowledge. An overview of the best-performing models
for about twenty application areas is then presented, e.g., question answering,
translation, story generation, dialog systems, generating images from text,
etc. For each application area, the strengths and weaknesses of current models
are discussed, and an outlook on further developments is given. In addition,
links are provided to freely available program code. A concluding chapter
summarizes the economic opportunities, mitigation of risks, and potential
developments of AI.
