  Large foundation models, including large language models (LLMs), vision
transformers (ViTs), diffusion, and LLM-based multimodal models, are
revolutionizing the entire machine learning lifecycle, from training to
deployment. However, the substantial advancements in versatility and
performance these models offer come at a significant cost in terms of hardware
resources. To support the growth of these large models in a scalable and
environmentally sustainable way, there has been a considerable focus on
developing resource-efficient strategies. This survey delves into the critical
importance of such research, examining both algorithmic and systemic aspects.
It offers a comprehensive analysis and valuable insights gleaned from existing
literature, encompassing a broad array of topics from cutting-edge model
architectures and training/serving algorithms to practical system designs and
implementations. The goal of this survey is to provide an overarching
understanding of how current approaches are tackling the resource challenges
posed by large foundation models and to potentially inspire future
breakthroughs in this field.
