  Large Language Models (LLMs) are consistently improving at increasingly
realistic software engineering (SE) tasks. In real-world software stacks,
significant SE effort is spent developing foundational system software like the
Linux kernel. Unlike application-level software, a systems codebase like Linux
is multilingual (low-level C/Assembly/Bash/Rust); gigantic (>20 million lines);
critical (impacting billions of devices worldwide), and highly concurrent
(involving complex multi-threading). To evaluate if ML models are useful while
developing such large-scale systems-level software, we introduce kGym (a
platform) and kBench (a dataset). The kGym platform provides a SE environment
for large-scale experiments on the Linux kernel, including compiling and
running kernels in parallel across several virtual machines, detecting
operations and crashes, inspecting logs, and querying and patching the code
base. We use kGym to facilitate evaluation on kBench, a crash resolution
benchmark drawn from real-world Linux kernel bugs. An example bug in kBench
contains crashing stack traces, a bug-reproducer file, a developer-written fix,
and other associated data. To understand current performance, we conduct
baseline experiments by prompting LLMs to resolve Linux kernel crashes. Our
initial evaluations reveal that the best performing LLM achieves 0.72% and
5.38% in the unassisted and assisted (i.e., buggy files disclosed to the model)
settings, respectively. These results highlight the need for further research
to enhance model performance in SE tasks. Improving performance on kBench
requires models to master new learning skills, including understanding the
cause of crashes and repairing faults, writing memory-safe and hardware-aware
code, and understanding concurrency. As a result, this work opens up multiple
avenues of research at the intersection of machine learning and systems
software.
