  Parallel programs in high performance computing (HPC) continue to grow in
complexity and scale in the exascale era. The diversity in hardware and
parallel programming models make developing, optimizing, and maintaining
parallel software even more burdensome for developers. One way to alleviate
some of these burdens is with automated development and analysis tools. Such
tools can perform complex and/or remedial tasks for developers that increase
their productivity and decrease the chance for error. Until recently, such
tools for code development and performance analysis have been limited in the
complexity of tasks they can perform, especially for parallel programs.
However, with recent advancements in language modeling, and the availability of
large amounts of open-source code related data, these tools have started to
utilize predictive language models to automate more complex tasks. In this
paper, we show how large language models (LLMs) can be applied to tasks
specific to high performance and scientific codes. We introduce a new dataset
of HPC and scientific codes and use it to fine-tune several pre-trained models.
We compare several pre-trained LLMs on HPC-related tasks and introduce a new
model, HPC-Coder, fine-tuned on parallel codes. In our experiments, we show
that this model can auto-complete HPC functions where generic models cannot,
decorate for loops with OpenMP pragmas, and model performance changes in
scientific application repositories as well as programming competition
solutions.
