  Automating software development processes through the orchestration of GitHub
Action workflows has revolutionized the efficiency and agility of software
delivery pipelines. This paper presents a detailed investigation into the use
of Large Language Models (LLMs) specifically, GPT 3.5 and GPT 4 to generate and
evaluate GitHub Action workflows for DevOps tasks. Our methodology involves
data collection from public GitHub repositories, prompt engineering for LLM
utilization, and evaluation metrics encompassing exact match scores, BLEU
scores, and a novel DevOps Aware score. The research scrutinizes the
proficiency of GPT 3.5 and GPT 4 in generating GitHub workflows, while
assessing the influence of various prompt elements in constructing the most
efficient pipeline. Results indicate substantial advancements in GPT 4,
particularly in DevOps awareness and syntax correctness. The research
introduces a GitHub App built on Probot, empowering users to automate workflow
generation within GitHub ecosystem. This study contributes insights into the
evolving landscape of AI-driven automation in DevOps practices.
