  Writing declarative models has numerous benefits, ranging from automated
reasoning and correction of design-level properties before systems are built,
to automated testing and debugging of their implementations after they are
built. Alloy is a declarative modeling language that is well-suited for
verifying system designs. A key strength of Alloy is its scenario-finding
toolset, the Analyzer, which allows users to explore all valid scenarios that
adhere to the model's constraints up to a user-provided scope. However, even
with visualized scenarios, it is difficult to write correct Alloy models. To
address this, a growing body of work explores different techniques for
debugging Alloy models. In order to develop and evaluate these techniques in an
effective manor, this paper presents an empirical study of over 97,000 models
written by novice users trying to learn Alloy. We investigate how users write
both correct and incorrect models in order to produce a comprehensive benchmark
for future use as well as a series of observations to guide debugging and
educational efforts for Alloy model development.
