  Large language models (LLMs) have shown remarkable progress in code
generation, but their generated code often suffers from inefficiency, resulting
in longer execution times and higher memory consumption. To address this issue,
we propose Self Optimization based on OverheAd Profile (SOAP), a
self-optimization framework that utilizes execution overhead profiles to
improve the efficiency of LLM-generated code. SOAP first generates code using
an LLM, then executes it locally to capture execution time and memory usage
profiles. These profiles are fed back to the LLM, which then revises the code
to reduce overhead. To evaluate the effectiveness of SOAP, we conduct extensive
experiments on the EffiBench, HumanEval, and MBPP with 16 open-source and 6
closed-source models. Our evaluation results demonstrate that through iterative
self-optimization, SOAP significantly enhances the efficiency of LLM-generated
code. For example, the execution time (ET) of StarCoder2-15B for the EffiBench
decreases from 0.93 (s) to 0.12 (s) which reduces 87.1% execution time
requirement compared with the initial code. The total memory usage (TMU) of
StarCoder2-15B also decreases from 22.02 (Mb*s) to 2.03 (Mb*s), which decreases
90.8% total memory consumption during the execution process. The source code of
SOAP was released in https://github.com/huangd1999/SOAP.
