  GitHub workflows or GitHub CI is a popular continuous integration platform
that enables developers to automate various software engineering tasks by
specifying them as workflows, i.e., YAML files with a list of jobs. However,
engineering valid workflows is tedious. They are also prone to severe security
issues, which can result in supply chain vulnerabilities. Recent advancements
in Large Language Models (LLMs) have demonstrated their effectiveness in
various software development tasks. However, GitHub workflows differ from
regular programs in both structure and semantics. We perform the first
comprehensive study to understand the effectiveness of LLMs on five
workflow-related tasks with different levels of prompts. We curated a set of
$\sim$400K workflows and generated prompts with varying detail. We also
fine-tuned LLMs on GitHub workflow tasks. Our evaluation of three
state-of-the-art LLMs and their fine-tuned variants revealed various
interesting findings on the current effectiveness and drawbacks of LLMs.
