  Transformers have become the de-facto standard model in artificial
intelligence since 2017 despite numerous shortcomings ranging from energy
inefficiency to hallucinations. Research has made a lot of progress in
improving elements of transformers, and, more generally, deep learning
manifesting in many proposals for architectures, layers, optimization
objectives, and optimization techniques. For researchers it is difficult to
keep track of such developments on a broader level. We provide a comprehensive
overview of the many important, recent works in these areas to those who
already have a basic understanding of deep learning. Our focus differs from
other works, as we target specifically novel, alternative potentially
disruptive approaches to transformers as well as successful ideas of recent
deep learning. We hope that such a holistic and unified treatment of
influential, recent works and novel ideas helps researchers to form new
connections between diverse areas of deep learning. We identify and discuss
multiple patterns that summarize the key strategies for successful innovations
over the last decade as well as works that can be seen as rising stars.
Especially, we discuss attempts on how to improve on transformers covering
(partially) proven methods such as state space models but also including
far-out ideas in deep learning that seem promising despite not achieving
state-of-the-art results. We also cover a discussion on recent state-of-the-art
models such as OpenAI's GPT series and Meta's LLama models and, Google's Gemini
model family.
