  Although repeatability and reproducibility are essential in science, failed
attempts to replicate results across diverse fields made some scientists argue
for a reproducibility crisis. In response, several high-profile venues within
computing established artifact evaluation tracks, a systematic procedure for
evaluating and badging research artifacts, with an increasing number of
artifacts submitted. This study compiles recent artifact evaluation procedures
and guidelines to show how artifact evaluation in distributed systems research
lags behind other computing disciplines and/or is less unified and more
complex. We further argue that current artifact assessment criteria are
uncoordinated and insufficient for the unique challenges of distributed systems
research. We examine the current state of the practice for artifacts and their
evaluation to provide recommendations to assist artifact authors, reviewers,
and track chairs. We summarize the recommendations and best practices as
checklists for artifact authors and evaluation committees. Although our
recommendations alone will not resolve the repeatability and reproducibility
crisis, we want to start a discussion in our community to increase the number
of submitted artifacts and their quality over time.
