  In July 2023, New York City became the first jurisdiction globally to mandate
bias audits for commercial algorithmic systems, specifically for automated
employment decisions systems (AEDTs) used in hiring and promotion. Local Law
144 (LL 144) requires AEDTs to be independently audited annually for race and
gender bias, and the audit report must be publicly posted. Additionally,
employers are obligated to post a transparency notice with the job listing. In
this study, 155 student investigators recorded 391 employers' compliance with
LL 144 and the user experience for prospective job applicants. Among these
employers, 18 posted audit reports and 13 posted transparency notices. These
rates could potentially be explained by a significant limitation in the
accountability mechanisms enacted by LL 144. Since the law grants employers
substantial discretion over whether their system is in scope of the law, a null
result cannot be said to indicate non-compliance, a condition we call ``null
compliance." Employer discretion may also explain our finding that nearly all
audits reported an impact factor over 0.8, a rule of thumb often used in
employment discrimination cases. We also find that the benefit of LL 144 to
ordinary job seekers is limited due to shortcomings in accessibility and
usability. Our findings offer important lessons for policy-makers as they
consider regulating algorithmic systems, particularly the degree of discretion
to grant to regulated parties and the limitations of relying on transparency
and end-user accountability.
