  Recently, language models (LMs), especially large language models (LLMs),
have revolutionized the field of deep learning. Both encoder-decoder models and
prompt-based techniques have shown immense potential for natural language
processing and code-based tasks. Over the past several years, many research
labs and institutions have invested heavily in high-performance computing,
approaching or breaching exascale performance levels. In this paper, we posit
that adapting and utilizing such language model-based techniques for tasks in
high-performance computing (HPC) would be very beneficial. This study presents
our reasoning behind the aforementioned position and highlights how existing
ideas can be improved and adapted for HPC tasks.
