  Algorithm audits are powerful tools for studying black-box systems. While
very effective in examining technical components, the method stops short of a
sociotechnical frame, which would also consider users as an integral and
dynamic part of the system. Addressing this gap, we propose the concept of
sociotechnical auditing: auditing methods that evaluate algorithmic systems at
the sociotechnical level, focusing on the interplay between algorithms and
users as each impacts the other. Just as algorithm audits probe an algorithm
with varied inputs and observe outputs, a sociotechnical audit (STA)
additionally probes users, exposing them to different algorithmic behavior and
measuring resulting attitudes and behaviors. To instantiate this method, we
develop Intervenr, a platform for conducting browser-based, longitudinal
sociotechnical audits with consenting, compensated participants. Intervenr
investigates the algorithmic content users encounter online and coordinates
systematic client-side interventions to understand how users change in
response. As a case study, we deploy Intervenr in a two-week sociotechnical
audit of online advertising (N=244) to investigate the central premise that
personalized ad targeting is more effective on users. In the first week, we
collect all browser ads delivered to users, and in the second, we deploy an
ablation-style intervention that disrupts normal targeting by randomly pairing
participants and swapping all their ads. We collect user-oriented metrics
(self-reported ad interest and feeling of representation) and
advertiser-oriented metrics (ad views, clicks, and recognition) throughout,
along with a total of over 500,000 ads. Our STA finds that targeted ads indeed
perform better with users, but also that users begin to acclimate to different
ads in only a week, casting doubt on the primacy of personalized ad targeting
given the impact of repeated exposure.
