  There is a growing need to gain insight into language model capabilities that
relate to sensitive topics, such as bioterrorism or cyberwarfare. However,
traditional open source benchmarks are not fit for the task, due to the
associated practice of publishing the correct answers in human-readable form.
At the same time, enforcing mandatory closed-quarters evaluations might stifle
development and erode trust. In this context, we propose hashmarking, a
protocol for evaluating language models in the open without having to disclose
the correct answers. In its simplest form, a hashmark is a benchmark whose
reference solutions have been cryptographically hashed prior to publication.
Following an overview of the proposed evaluation protocol, we go on to assess
its resilience against traditional attack vectors (e.g. rainbow table attacks),
as well as against failure modes unique to increasingly capable generative
models.
