  Algorithmic decision making is now widespread, ranging from health care
allocation to more common actions such as recommendation or information
ranking. The aim to audit these algorithms has grown alongside. In this paper,
we focus on external audits that are conducted by interacting with the user
side of the target algorithm, hence considered as a black box. Yet, the legal
framework in which these audits take place is mostly ambiguous to researchers
developing them: on the one hand, the legal value of the audit outcome is
uncertain; on the other hand the auditors' rights and obligations are unclear.
The contribution of this paper is to articulate two canonical audit forms to
law, to shed light on these aspects: 1) the first audit form (we coin the Bobby
audit form) checks a predicate against the algorithm, while the second
(Sherlock) is more loose and opens up to multiple investigations. We find that:
Bobby audits are more amenable to prosecution, yet are delicate as operating on
real user data. This can lead to reject by a court (notion of admissibility).
Sherlock audits craft data for their operation, most notably to build
surrogates of the audited algorithm. It is mostly used for acts for
whistleblowing, as even if accepted as a proof, the evidential value will be
low in practice. 2) these two forms require the prior respect of a proper right
to audit, granted by law or by the platform being audited; otherwise the
auditor will be also prone to prosecutions regardless of the audit outcome.
This article thus highlights the relation of current audits with law, in order
to structure the growing field of algorithm auditing.
