  Mathematical optimization is now widely regarded as an indispensable modeling
and solution tool for the design of wireless communications systems. While
optimization has played a significant role in the revolutionary progress in
wireless communication and networking technologies from 1G to 5G and onto the
future 6G, the innovations in wireless technologies have also substantially
transformed the nature of the underlying mathematical optimization problems
upon which the system designs are based and have sparked significant
innovations in the development of methodologies to understand, to analyze, and
to solve those problems. In this paper, we provide a comprehensive survey of
recent advances in mathematical optimization theory and algorithms for wireless
communication system design. We begin by illustrating common features of
mathematical optimization problems arising in wireless communication system
design. We discuss various scenarios and use cases and their associated
mathematical structures from an optimization perspective. We then provide an
overview of recently developed optimization techniques in areas ranging from
nonconvex optimization, global optimization, and integer programming, to
distributed optimization and learning-based optimization. The key to successful
solution of mathematical optimization problems is in carefully choosing or
developing suitable algorithms (or neural network architectures) that can
exploit the underlying problem structure. We conclude the paper by identifying
several open research challenges and outlining future research directions.
