  The rise of capabilities expressed by large language models has been quickly
followed by the integration of the same complex systems into application level
logic. Algorithms, programs, systems, and companies are built around structured
prompting to black box models where the majority of the design and
implementation lies in capturing and quantifying the `agent mode'. The standard
way to shape a closed language model is to prime it for a specific task with a
tailored prompt, often initially handwritten by a human. The textual prompts
co-evolve with the codebase, taking shape over the course of project life as
artifacts which must be reviewed and maintained, just as the traditional code
files might be. Unlike traditional code, we find that prompts do not receive
effective static testing and linting to prevent runtime issues. In this work,
we present a novel dataset called PromptSet, with more than 61,000 unique
developer prompts used in open source Python programs. We perform analysis on
this dataset and introduce the notion of a static linter for prompts. Released
with this publication is a HuggingFace dataset and a Github repository to
recreate collection and processing efforts, both under the name
\texttt{pisterlabs/promptset}.
