  Large language models (LLMs) have been successfully applied to software
engineering tasks, including program repair. However, their application in
search-based techniques such as Genetic Improvement (GI) is still largely
unexplored. In this paper, we evaluate the use of LLMs as mutation operators
for GI to improve the search process. We expand the Gin Java GI toolkit to call
OpenAI's API to generate edits for the JCodec tool. We randomly sample the
space of edits using 5 different edit types. We find that the number of patches
passing unit tests is up to 75% higher with LLM-based edits than with standard
Insert edits. Further, we observe that the patches found with LLMs are
generally less diverse compared to standard edits. We ran GI with local search
to find runtime improvements. Although many improving patches are found by
LLM-enhanced GI, the best improving patch was found by standard GI.
