  In recent years, there has been remarkable progress in leveraging Language
Models (LMs), encompassing Pre-trained Language Models (PLMs) and Large-scale
Language Models (LLMs), within the domain of mathematics. This paper conducts a
comprehensive survey of mathematical LMs, systematically categorizing pivotal
research endeavors from two distinct perspectives: tasks and methodologies. The
landscape reveals a large number of proposed mathematical LLMs, which are
further delineated into instruction learning, tool-based methods, fundamental
CoT techniques, and advanced CoT methodologies. In addition, our survey entails
the compilation of over 60 mathematical datasets, including training datasets,
benchmark datasets, and augmented datasets. Addressing the primary challenges
and delineating future trajectories within the field of mathematical LMs, this
survey is positioned as a valuable resource, poised to facilitate and inspire
future innovation among researchers invested in advancing this domain.
