  Pre-trained large language models (LLMs) have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks and to be appropriately specialized to
particular domains. Here, we target bioinformatics due to the amount of domain
knowledge, algorithms, and data operations this discipline requires. We present
BioCoder, a benchmark developed to evaluate LLMs in generating
bioinformatics-specific code. BioCoder spans much of the field, covering
cross-file dependencies, class declarations, and global variables. It
incorporates 1,026 Python functions and 1,243 Java methods extracted from
GitHub, along with 253 examples from the Rosalind Project, all pertaining to
bioinformatics. Using topic modeling, we show that the overall coverage of the
included code is representative of the full spectrum of bioinformatics
calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We
have applied it to evaluate various models including InCoder, CodeGen,
CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-
4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our
training dataset can enhance the performance on our testing benchmark (by >15%
in terms of Pass@K under certain prompt configurations and always >3%). The
results highlight two key aspects of successful models: (1) Successful models
accommodate a long prompt (> 2,600 tokens) with full context, including
functional dependencies. (2) They contain domain-specific knowledge of
bioinformatics, beyond just general coding capability. This is evident from the
performance gain of GPT-3.5/4 compared to the smaller models on our benchmark
(50% vs. up to 25%). Availability and implementation: Code is available at:
https://github.com/gersteinlab/biocoder and https://biocoder-benchmark.
github.io/.
